286 Commits
v1.19 ... v1.20

Author SHA1 Message Date
Darshit Shah
e55916e5de Prepare NEWS for new release 2018-11-13 16:24:17 +01:00
Darshit Shah
c4a6b89b2e * configure.ac: gnulib now expects autoconf >=2.63 2018-11-13 16:20:19 +01:00
Darshit Shah
3d4e1f47a1 * gnulib: Update library 2018-11-13 16:20:19 +01:00
Jay Satiro
61271d87f6 * src/init.c: Stop freeing the pointer returned by ws_mypath()
.. since ws_mypath() saves the address it returns in a static pointer
for reuse, to also be returned in later calls.
2018-11-13 15:51:51 +01:00
Darshit Shah
2bc2d2f803 * src/ftp.c(ftp_retrieve_glob): Honor {accept,reject}-regex switches as well 2018-11-13 15:51:51 +01:00
Darshit Shah
8c741da256 * src/ftp.c (ftp_retrieve_glob): Refactor to prevent looping over listing multiple times 2018-11-13 15:51:51 +01:00
Tim Rühsen
d3383d619b * .gitlab-ci.yml: Split into GnuTLS and OpenSSL build 2018-11-11 18:43:46 +01:00
Tim Rühsen
62dced2f43 * Makefile.am: dist clean po/stamp-po 2018-11-11 17:51:37 +01:00
Tim Rühsen
04116409a8 Remove auto-generated files from po/ 2018-11-11 16:58:09 +01:00
Tim Rühsen
7b8325b416 Add VPATH build 2018-11-11 16:57:41 +01:00
Tim Rühsen
11fad3fa72 Revert "Bail out on unexpected 416 server errors"
This reverts commit 6f3b995993.

The code is obviously wrong, see https://savannah.gnu.org/bugs/?54963
Also, the example from the original post doesn't work any more.
With other words, the broken server behavior has been fixed meanwhile.
2018-11-09 16:16:43 +01:00
Rosen Penev
a3643c6076 openssl: Do not use engines when OpenSSL does not support
* src/openssl.c: Check for OPENSSL_NO_ENGINE before
 including openssl/engine.h and before calling ENGINE_load_builtin_engines()

Fixes compilation with no engines compiled.

Copyright-paperwork-exempt: Yes
Signed-off-by: Rosen Penev <rosenp@gmail.com>
2018-11-09 16:01:51 +01:00
Tim Rühsen
ed23e682da Fix HTTPS Perl tests
* tests/SSLTest.pm: Rename server cert and key file
* tests/Test-https*.px: Fix and remove OpenSSL hard-coding
* tests/certs/create-certs.sh: Script to generate test files
* tests/certs/*-template.txt: GnuTLS template files for certs and crl
* tests/certs/*.pem: Keys, certs, crls
* tests/certs/README: Removed commands, link to create-certs.sh
2018-11-09 15:47:53 +01:00
Kapus, Timotej
6d7cd9313c Replace some loops with string.h functions
* src/init.c: Replace loop with strspn
* src/url.c: Replace loop with strrchr

Copyright-paperwork-exempt: Yes
2018-10-28 10:36:46 +01:00
Luiz Angelo Daros de Luca
7903dbc9d8 * .gitmodules: Use https:// instead of git:// for gnulib
git:// does not work over http proxy

Signed-off-by: Luiz Angelo Daros de Luca <luizluca@gmail.com>
Copyright-paperwork-exempt: Yes
2018-10-26 22:55:44 +02:00
Luiz Angelo Daros de Luca
fd85ac9cc6 * src/host.c (sufmatch): Fix dot-prefixed domain matching
Current sufmatch does not match when domain is dot-prefixed.
The example of no_proxy in man (.mit.edu) does use a dot-prefixed
domain.

Signed-off-by: Luiz Angelo Daros de Luca <luizluca@gmail.com>
Copyright-paperwork-exempt: Yes
2018-10-26 22:54:26 +02:00
Tim Rühsen
21daa24e72 * src/convert.c (convert_links): Fix fallthrough 2018-10-26 22:52:41 +02:00
Darshit Shah
6d6679ba86 * bootstrap: Update script from gnulib 2018-10-22 16:45:58 +02:00
Darshit Shah
d18f78b86b * gnulib: Update library 2018-10-22 16:45:58 +02:00
Tim Rühsen
85310bfa58 * .lgtm.yml: New file to add LGTM to Gitlab.com CI 2018-10-19 12:32:01 +02:00
Tim Rühsen
b29854528a * configure.ac: Fix build issue with libgpgme 2018-10-16 10:31:54 +02:00
Tim Rühsen
3d518f125c * fuzz/*_fuzzer.in/*: Update fuzzer corpora 2018-10-14 15:15:49 +02:00
Nikos Mavrogiannopoulos
c11cc83d9e Enable post-handshake auth under gnutls on TLS1.3 2018-10-08 15:55:48 +02:00
Tim Rühsen
0727b8f3a9 * src/http.c (resp_new): Fix code to avoid false positive by clang 2018-09-20 14:59:06 +02:00
Tim Rühsen
88a49c1e41 * src/convert.c (convert_links): Fix code to avoid false positive by clang 2018-09-20 14:58:27 +02:00
Tim Rühsen
02afe1e41c Add support for PCRE2 pattern matching
* configure.ac: Check for libpcre2-8
* src/init.c (choices): Test for HAVE_LIBPCRE2
* src/main.c (main): Set regex compile and match functions
* src/options.h: Test for HAVE_LIBPCRE2
* src/utils.c: Include pcre2.h, add functions
  compile_pcre2_regex() and match_pcre2_regex()
* src/utils.h: Declare compile_pcre2_regex() and match_pcre2_regex()

Fixes #54677
Reported-by: Noël Köthe
2018-09-19 16:22:25 +02:00
Tim Rühsen
79be99aff4 Add . to perl path for all perl tests
* tests/*.px: Add -I . to the shebang

This allows perl test to be run from tests/ directory, e.g. via
  ./Test--post-file.px
2018-09-07 10:48:14 +02:00
Tomas Hozza
2bbdfd76da Add TLS 1.3 support for GnuTLS
* doc/wget.texi: Add "TLSv1_3" to --secure-protocol
* src/gnutls.c (set_prio_default): Use GNUTLS_TLS1_3 where needed

Wget currently allows specifying "TLSv1_3" as the parameter for
--secure-protocol option. However it is only implemented for OpenSSL
and in case wget is compiled with GnuTLS, it causes wget to abort with:
GnuTLS: unimplemented 'secure-protocol' option value 6

GnuTLS contains TLS 1.3 implementation since version 3.6.3 [1]. However
currently it must be enabled explicitly in the application of it to be
used. This will change after the draft is finalized. [2] However for
the time being, I enabled it explicitly in case "TLSv1_3" is used with
--secure-protocol.

I also fixed man page to contain "TLSv1_3" in all listings of available
parameters for --secure-protocol

[1] https://lists.gnupg.org/pipermail/gnutls-devel/2018-July/008584.html
[2] https://nikmav.blogspot.com/2018/05/gnutls-and-tls-13.html

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-09-07 09:56:02 +02:00
Tomas Korbar
7ddcebd61e Avoid creating empty wget-log when using -O and -q in background
* src/log.c (check_redirect_output): Check for quiet mode
2018-08-29 12:34:03 +02:00
Tomas Hozza
2f451dbf4e * src/warc.c (warc_write_cdx_record): Fix RESOURCE LEAK found by Coverity
Error: RESOURCE_LEAK (CWE-772): - REAL ERROR
wget-1.19.5/src/warc.c:1376: alloc_fn: Storage is returned from allocation function "url_escape".
wget-1.19.5/src/url.c:284:3: alloc_fn: Storage is returned from allocation function "url_escape_1".
wget-1.19.5/src/url.c:255:3: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/url.c:255:3: var_assign: Assigning: "newstr" = "xmalloc(newlen + 1)".
wget-1.19.5/src/url.c:258:3: var_assign: Assigning: "p2" = "newstr".
wget-1.19.5/src/url.c:275:3: return_alloc: Returning allocated memory "newstr".
wget-1.19.5/src/url.c:284:3: return_alloc_fn: Directly returning storage allocated by "url_escape_1".
wget-1.19.5/src/warc.c:1376: var_assign: Assigning: "redirect_location" = storage returned from "url_escape(redirect_location)".
wget-1.19.5/src/warc.c:1381: noescape: Resource "redirect_location" is not freed or pointed-to in "fprintf".
wget-1.19.5/src/warc.c:1387: leaked_storage: Returning without freeing "redirect_location" leaks the storage that it points to.
\# 1385|     fflush (warc_current_cdx_file);
\# 1386|
\# 1387|->   return true;
\# 1388|   }
\# 1389|

url_escape() really returns a newly allocated memory and it leaks when the warc_write_cdx_record() returns. The memory returned from url_escape() is usually stored in a temporary variable in other parts of the project and then freed. I took the same approach.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-08-27 13:25:34 +02:00
Tomas Hozza
8b451f9f21 * src/warc.c (warc_write_start_record): Fix potential RESOURCE LEAK
In warc_write_start_record() function, the reutrn value of dup() is
directly used in gzdopen() call and not stored anywhere. However the
zlib documentation says that "The duplicated descriptor should be saved
to avoid a leak, since gzdopen does not close fd if it fails." [1].
This change stores the FD in a variable and closes it in case gzopen()
fails.

[1] https://www.zlib.net/manual.html

Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/warc.c:217: open_fn: Returning handle opened by "dup".
wget-1.19.5/src/warc.c:217: leaked_handle: Failing to save or close handle opened by "dup(fileno(warc_current_file))" leaks it.
\#  215|
\#  216|         /* Start a new GZIP stream. */
\#  217|->       warc_current_gzfile = gzdopen (dup (fileno (warc_current_file)), "wb9");
\#  218|         warc_current_gzfile_uncompressed_size = 0;
\#  219|

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-08-27 13:25:07 +02:00
Tomas Hozza
c045cdded4 * src/utils.c (open_stat): Fix RESOURCE LEAK found by Coverity
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/utils.c:914: open_fn: Returning handle opened by "open". [Note: The source code implementation of the function has been overridden by a user model.]
wget-1.19.5/src/utils.c:914: var_assign: Assigning: "fd" = handle returned from "open(fname, flags, mode)".
wget-1.19.5/src/utils.c:921: noescape: Resource "fd" is not freed or pointed-to in "fstat". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/utils.c:924: leaked_handle: Handle variable "fd" going out of scope leaks the handle.
\#  922|     {
\#  923|       logprintf (LOG_NOTQUIET, _("Failed to stat file %s, error: %s\n"), fname, strerror(errno));
\#  924|->     return -1;
\#  925|     }
\#  926|   #if !(defined(WINDOWS) || defined(__VMS))

This seems to be a real issue, since the opened file descriptor in "fd"
would leak. There is also additional check below the "fstat" call, which
closes the opened "fd".

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-08-27 13:24:46 +02:00
Tomas Hozza
dfef92bac3 * src/http.c (http_loop): Fix RESOURCE LEAK found by Coverity
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/http.c:4486: alloc_fn: Storage is returned from allocation function "url_string".
wget-1.19.5/src/url.c:2248:3: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/url.c:2248:3: var_assign: Assigning: "result" = "xmalloc(size)".
wget-1.19.5/src/url.c:2248:3: var_assign: Assigning: "p" = "result".
wget-1.19.5/src/url.c:2250:3: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2253:7: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2257:11: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2264:3: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2270:7: identity_transfer: Passing "p" as argument 1 to function "number_to_string", which returns an offset off that argument.
wget-1.19.5/src/utils.c:1776:11: var_assign_parm: Assigning: "p" = "buffer".
wget-1.19.5/src/utils.c:1847:3: return_var: Returning "p", which is a copy of a parameter.
wget-1.19.5/src/url.c:2270:7: noescape: Resource "p" is not freed or pointed-to in function "number_to_string".
wget-1.19.5/src/utils.c:1774:25: noescape: "number_to_string(char *, wgint)" does not free or save its parameter "buffer".
wget-1.19.5/src/url.c:2270:7: var_assign: Assigning: "p" = "number_to_string(p, url->port)".
wget-1.19.5/src/url.c:2273:3: noescape: Resource "p" is not freed or pointed-to in function "full_path_write".
wget-1.19.5/src/url.c:1078:47: noescape: "full_path_write(struct url const *, char *)" does not free or save its parameter "where".
wget-1.19.5/src/url.c:2287:3: return_alloc: Returning allocated memory "result".
wget-1.19.5/src/http.c:4486: var_assign: Assigning: "hurl" = storage returned from "url_string(u, URL_AUTH_HIDE_PASSWD)".
wget-1.19.5/src/http.c:4487: noescape: Resource "hurl" is not freed or pointed-to in "logprintf".
wget-1.19.5/src/http.c:4513: leaked_storage: Variable "hurl" going out of scope leaks the storage it points to.
\# 4511|               {
\# 4512|                 printwhat (count, opt.ntry);
\# 4513|->               continue;
\# 4514|               }
\# 4515|             else

There are two conditional branches, which call continue, without freeing memory potentially allocated and pointed to by"hurl" pointer. In fase "!opt.verbose" is True and some of the appropriate conditions in the following if/else if construction, in which "continue" is called, are also true, then the memory allocated to "hurl" will leak.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-08-27 13:24:24 +02:00
Tomas Hozza
b8be904ac7 * src/http.c (check_auth): Fix RESOURCE LEAK found by Coverity
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/http.c:2434: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/http.c:2434: var_assign: Assigning: "auth_stat" = storage returned from "xmalloc(4UL)".
wget-1.19.5/src/http.c:2446: noescape: Resource "auth_stat" is not freed or pointed-to in "create_authorization_line".
wget-1.19.5/src/http.c:5203:70: noescape: "create_authorization_line(char const *, char const *, char const *, char const *, char const *, _Bool *, uerr_t *)" does not free or save its parameter "auth_err".
wget-1.19.5/src/http.c:2476: leaked_storage: Variable "auth_stat" going out of scope leaks the storage it points to.
\# 2474|                 /* Creating the Authorization header went wrong */
\# 2475|               }
\# 2476|->         }
\# 2477|         else
\# 2478|           {

Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/http.c:2431: alloc_fn: Storage is returned from allocation function "url_full_path".
wget-1.19.5/src/url.c:1105:19: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/url.c:1105:19: var_assign: Assigning: "full_path" = "xmalloc(length + 1)".
wget-1.19.5/src/url.c:1107:3: noescape: Resource "full_path" is not freed or pointed-to in function "full_path_write".
wget-1.19.5/src/url.c:1078:47: noescape: "full_path_write(struct url const *, char *)" does not free or save its parameter "where".
wget-1.19.5/src/url.c:1110:3: return_alloc: Returning allocated memory "full_path".
wget-1.19.5/src/http.c:2431: var_assign: Assigning: "pth" = storage returned from "url_full_path(u)".
wget-1.19.5/src/http.c:2446: noescape: Resource "pth" is not freed or pointed-to in "create_authorization_line".
wget-1.19.5/src/http.c:5203:40: noescape: "create_authorization_line(char const *, char const *, char const *, char const *, char const *, _Bool *, uerr_t *)" does not free or save its parameter "path".
wget-1.19.5/src/http.c:2476: leaked_storage: Variable "pth" going out of scope leaks the storage it points to.
\# 2474|                 /* Creating the Authorization header went wrong */
\# 2475|               }
\# 2476|->         }
\# 2477|         else
\# 2478|           {

Both "pth" and "auth_stat" are allocated in "check_auth()" function. These are used for creating the HTTP Authorization Request header via "create_authorization_line()" function. In case the creation went OK (auth_err == RETROK), then the memory previously allocated to "pth" and "auth_stat" is freed. However if the creation failed, then the memory is never freed and it leaks.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-08-27 13:23:52 +02:00
Tomas Hozza
b24351183e * src/ftp.c (getftp): Fix RESOURCE LEAK found by Coverity
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/ftp.c:1493: alloc_fn: Storage is returned from allocation function "fopen".
wget-1.19.5/src/ftp.c:1493: var_assign: Assigning: "fp" = storage returned from "fopen(con->target, "wb")".
wget-1.19.5/src/ftp.c:1811: leaked_storage: Variable "fp" going out of scope leaks the storage it points to.
\# 1809|     if (fp && !output_stream)
\# 1810|       fclose (fp);
\# 1811|->   return err;
\# 1812|   }
\# 1813|

It can happen, that "if (!output_stream || con->cmd & DO_LIST)" on line #1398 can be true, even though "output_stream != NULL". In this case a new file is opened to "fp". Later it may happen in the FTPS branch, that some error will occure and code will jump to label "exit_error". In "exit_error", the "fp" is closed only if "output_stream == NULL". However this may not be true as described earlier and "fp" leaks.

On line #1588, there is the following conditional free of "fp":

  /* Close the local file.  */
  if (!output_stream || con->cmd & DO_LIST)
    fclose (fp);

Therefore the conditional at the end of the function after "exit_error" label should be modified to:

  if (fp && (!output_stream || con->cmd & DO_LIST))
    fclose (fp);

This will ensure that "fp" does not leak in any case it sould be opened.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-08-27 13:20:48 +02:00
Tomas Hozza
2b2283d3e2 Don't limit the test suite HTTPS server to TLSv1
In Fedora, we are implementing crypto policies, in order to enhance the
security of user systems. This is done on the system level by global
configuration. It may happen that due to the active policy, only
TLSv1.2 or higher will be available in crypto libraries. While wget as
a client will by default determine the minimal TLS version supported by
both client and server, the HTTPS server implementation in testenv/
hardcodes use of TLSv1. As a result all HTTPS related tests fail in
case a more hardened crypto policy is set on the Fedora system.

This change removes the explicit TLS version setting and leaves the
determination of the minimal supported TLS version on the server and
client.

More information about Fedora change can be found here:
https://fedoraproject.org/wiki/Changes/StrongCryptoSettings

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2018-08-11 12:51:13 +02:00
Tim Rühsen
122a9f08a3 * src/gnutls.c (ssl_check_certificate): Fix grammar of error msg
Reported-by: Nicholas Sielicki
2018-06-13 20:34:24 +02:00
Tim Rühsen
333746f787 * fuzz/Makefile.am: Remove libtool LTLIB... from LDADD 2018-06-13 20:16:46 +02:00
Tim Rühsen
4fc69950da * src/http.c (http_loop): Fix --retry-on-host-error 2018-06-13 20:16:22 +02:00
ethus3h
e7979da9e8 Add new option --retry-on-host-error
* doc/wget.texi: Add docs for --retry-on-host-error
* src/http.c (http_loop): Add code for HOSTERR
* src/init.c: Add option --retry-on-host-error
* src/main.c: Likewise
* src/options.h: Add options.retry_on_host_error

Copyright-paperwork-exempt: Yes
2018-06-13 20:10:28 +02:00
Tim Rühsen
ad261f41ce Save original data to WARC file
* src/retr.c (write_data): Cleanup,
  (fd_read_body): Write to WARC before uncompressing

Fixes: #53968
2018-05-29 10:52:20 +02:00
Tim Rühsen
c88500fca8 * fuzz/get_ossfuzz_corpora: Speed up corpora download 2018-05-10 19:33:59 +02:00
Tim Rühsen
4188fcdced * src/main.c (print_version): Silence UBSAN message 2018-05-09 13:56:20 +02:00
Tim Rühsen
4bdb09d3a7 * src/utils.ci (file_exists_p): Fix stat(NULL,...) 2018-05-09 12:37:03 +02:00
Tim Rühsen
35f5f79ce1 * src/hsts.c (open_hsts_test_store): Fix unlink(NULL) 2018-05-09 12:29:39 +02:00
Tim Rühsen
3cbdc67c96 * src/hash.c: Silence UBSAN for hash functions 2018-05-09 12:16:51 +02:00
Tim Rühsen
cdaee00259 * fuzz/*_fuzzer.in: Update corpora from OSS-Fuzz 2018-05-09 11:45:22 +02:00
Tim Rühsen
ad2471425f * fuzz/get_ossfuzz_corpora: Fix path 2018-05-09 11:44:18 +02:00
Tim Rühsen
ace96e4412 * src/hsts.h: Fix header guard 2018-05-08 10:17:06 +02:00
Tim Rühsen
77286a2e03 * src/version.h: Add header guard 2018-05-08 10:10:44 +02:00
Tim Rühsen
7eff94e881 * src/host.c (wait_ares): Remove void assignment
Reported-by: Josef Moellers
2018-05-08 09:36:48 +02:00
Tim Rühsen
15a39093b8 Update NEWS file for new release 2018-05-06 18:38:29 +02:00
Tim Rühsen
1fc9c95ec1 Fix cookie injection (CVE-2018-0494)
* src/http.c (resp_new): Replace \r\n by space in continuation lines

Fixes #53763
 "Malicious website can write arbitrary cookie entries to cookie jar"

HTTP header parsing left the \r\n from continuation line intact.
The Set-Cookie code didn't check and could be tricked to write
\r\n into the cookie jar, allowing a server to generate cookies at will.
2018-05-06 18:24:58 +02:00
Tim Rühsen
f51936745a * tests/Test-https-weboftrust.px: Skip test, needs cert regen 2018-05-06 18:19:50 +02:00
Tim Rühsen
491c6914cb Fix make syntax-check
* cfg.mk: Add fuzzer reproducers to exception list
* po/POTFILES.in: Add src/spider.c
2018-05-06 17:44:37 +02:00
Tim Rühsen
a6452061f8 Fix HTTPS tests
* tests/Test-https-badcerts.px: Fix test return value
* tests/Test-https-crl.px: Likewise
* README: How to create certs with GnuTLS's certtool
* tests/certs/revokedcrl.pem: Recreated revocation
* tests/certs/server.crt: Recreated server cert with no expiry
* tests/certs/test-ca-cert.pem: Recreated CA cert with no expiry
2018-05-06 17:30:42 +02:00
Tim Rühsen
77cf701416 * src/init.c: Bring new --ciphers into right order in options array 2018-05-06 12:49:46 +02:00
Ander Juaristi
c4eb863299 * doc/wget.texi: Add description for --ciphers 2018-05-05 22:50:23 +02:00
Ander Juarist
b9c4cadd84 OpenSSL: Better seeding of PRNG
* src/openssl.c (init_prng): keep gathering entropy even though we
                              already have enough
   (ssl_connect_with_timeout_callback): reseed PRNG again just before
                                        the handshake

Reported-by: Jeffrey Walton <noloader@gmail.com>
2018-05-05 22:49:06 +02:00
Ander Juaristi
744671aac6 Enhance SSL/TLS security
This commit hardens SSL/TLS a bit more in the following ways:

 * Explicitly exclude NULL authentication and the 'MEDIUM' cipher list
   category. Ciphers in the 'HIGH' level are only considered - this
   includes all symmetric ciphers with key lengths larger than 128 bits,
   and some ('modern') 128-bit ciphers, such as AES in GCM mode.
 * Allow RSA key exchange by default, but exclude it when
   Perfect Forward Secrecy is desired (with --secure-protocol=PFS).
 * Introduce new option --ciphers to set the cipher list that the SSL/TLS
   engine will favor. This string is fed directly to the underlying TLS
   library (GnuTLS or OpenSSL) without further processing, and hence its
   format and syntax are directly dependent on the specific library.

Reported-by: Jeffrey Walton <noloader@gmail.com>
2018-05-05 22:49:06 +02:00
Tim Rühsen
26a50942d8 * src/netrc.c (parse_netrc_fp): Fix two memleaks 2018-04-28 20:50:30 +02:00
Tim Rühsen
a1c9018797 Add new fuzzer for the .netrc parser
* fuzz/wget_netrc_fuzzer.c: New fuzzer
* fuzz/wget_netrc_fuzzer.dict: Fuzzer dictionary
* fuzz/wget_netrc_fuzzer.in: Initial corpora
* src/ftp.c (getftp): Amend call to search_netrc()
* src/http.c (initialize_request): Likewise
* src/netrc.c: Cleanup, prepare code for fuzzing
* src/netrc.h: Cleanup
2018-04-28 20:49:57 +02:00
Tim Rühsen
734d0aee15 * src/utils.c (match_tail): Fix unsigned integer overflow 2018-04-27 12:56:25 +02:00
Tim Rühsen
7de006bade Add new fuzzer for the Set-Cookie parser
* fuzz/Makefile.am: Add wget_cookie_fuzzer
* fuzz/wget_cookie_fuzzer.c: New fuzzer
* fuzz/wget_cookie_fuzzer.dict: Fuzzers dictionary
* fuzz/wget_cookie_fuzzer.in: Initial corpora
2018-04-27 12:56:25 +02:00
Tim Rühsen
78838d761f Fix buffer overflow in CSS parser
* src/css-url.c (get_uri_string): Check input length
* fuzz/wget_css_fuzzer.repro/buffer-overflow-6600180399865856:
  Add reproducer corpus

Fixes OSS-Fuzz issue #8033.
This is a long standing bug affecting all versions <= 1.19.4.
2018-04-26 22:40:28 +02:00
Tim Rühsen
cb47f3aaa4 Fix buffer overflow in CSS parser
* src/css-url.c (get_urls_css): Check input string length
* fuzz/wget_css_fuzzer.repro/negative-size-param-5724866467594240:
  Add reproducer corpus

Fixes OSS-Fuzz issue #8032.
This is a long standing bug affecting all versions <= 1.19.4.
2018-04-26 21:25:28 +02:00
Tim Rühsen
acfd9b4d56 Exclude fuzz corpora from tarball
* fuzz/Makefile.am: Do not include corpora in tarball
* fuzz/main.c: SKIP if corpora directory isn't found (make check)

The fuzz corpora are thousands of files, not needed for a standard build
from a distribution tarball. The reproducers of former issues are being
included for regression testing.
2018-04-26 16:18:01 +02:00
Tim Rühsen
ceb5d2d794 * tests/Makefile.am: Add -I/src to AM_CPPFLAGS 2018-04-26 16:17:10 +02:00
Tim Rühsen
939dbb0ebb Add CSS slowness reproducer (fixed)
* fuzz/wget_css_fuzzer.repro/slowness-6275836549267456: New file

This file created an extreme CPU usage with the old CSS parser.
2018-04-26 16:07:46 +02:00
Tim Rühsen
caa08d7470 Update CSS grammar from 1.x to 2.2
* src/css-tokens.h: Add enums and fixate values
* src/css.l: Include config.h,
  ignore several compiler warnings,
  update the grammar to CSS 2.2

Fixes OSS-Fuzz issue #8010 (slowness issue).
This is a long standing bug affecting all versions <= 1.19.4.

Some crafted CSS input was extremely slow / CPU wasting, so it could
be used as a DOS attack against website scanning.

The code/grammar changes were backported from Wget2.x.
2018-04-26 13:10:39 +02:00
Tim Rühsen
76fb1fe6f6 * src/res.c (add_path): Fix memleak (parsing robots.txt)
Fixes OSS-Fuzz issue #8005.
This is a long standing bug affecting all versions <= 1.19.4.
2018-04-25 11:33:38 +02:00
Tim Rühsen
fe6d1247ad * src/ftp-ls.c (ftp_parse_winnt_ls): Fix integer overflow
Fixes OSS-Fuzz issue #7999.
This is a long standing bug affecting all versions <= 1.19.4.
2018-04-25 09:37:29 +02:00
Tim Rühsen
02325168ca Add new fuzzer for the URL parser
* fuzz/Makefile.am: Add wget_url_fuzzer
* fuzz/wget_url_fuzzer.c: New fuzzer
* fuzz/wget_url_fuzzer.in: Initial corpora
2018-04-24 21:36:06 +02:00
Tim Rühsen
93e5a97f25 Add new fuzzer for robots.txt parsing
* fuzz/Makefile.am: Add wget_robots_fuzzer
* fuzz/wget_robots_fuzzer.c: New fuzzer
* fuzz/wget_robots_fuzzer.in: Initial corpora
2018-04-24 11:47:49 +02:00
Tim Rühsen
36482a21ea * fuzz/README.md: Add CFLAGS for undefined sanitizer 2018-04-24 11:30:06 +02:00
Tim Rühsen
7ee3ad1c48 * src/ftp-ls.c (ftp_parse_winnt_ls): Fix integer overflow 2018-04-24 11:11:47 +02:00
Tim Rühsen
79c1f333dc * src/ftp-ls.c (ftp_parse_vms_ls): Fix integer overflow by left shift 2018-04-24 11:05:52 +02:00
Tim Rühsen
d8365b0607 * src/ftp-ls.c (ftp_parse_unix_ls): Fix integer overflow in date parsing 2018-04-24 10:55:29 +02:00
Tim Rühsen
b0f802c46c * src/ftp-ls.c (ftp_parse_winnt_ls): Fix heap-buffer-overflow
Fixes OSS-Fuzz issue #7931.
This is a long standing bug affecting all versions <= 1.19.4.
2018-04-22 12:45:51 +02:00
Tim Rühsen
96c64a859d * src/ftp-ls.c (ftp_parse_winnt_ls): Fix heap-buffer-overflow
Fixes OSS-Fuzz issue #7930.
This is a long standing bug affecting all versions <= 1.19.4.
2018-04-22 11:33:35 +02:00
Tim Rühsen
2269cc2f1b * fuzz/wget_ftpls_fuzzer.in: Update corpora 2018-04-22 00:29:47 +02:00
Tim Rühsen
7d3da08537 * src/ftp-ls.c (eat_carets): Fix heap-buffer-overflow 2018-04-21 23:48:01 +02:00
Tim Rühsen
2b61c46183 * src/ftp-ls.c (ftp_parse_winnt_ls): Fix memleak 2018-04-21 22:52:01 +02:00
Tim Rühsen
f0d715b264 * src/ftp-ls.c (ftp_parse_vms_ls): Fix heap-buffer-overflow 2018-04-21 22:47:17 +02:00
Tim Rühsen
b3ff8ce3d5 * src/ftp-ls.c (ftp_parse_vms_ls): Fix heap-buffer-overflow 2018-04-21 22:45:03 +02:00
Tim Rühsen
c7014fbaea * src/ftp-ls.c (ftp_parse_vms_ls): Fix memleak 2018-04-21 22:42:30 +02:00
Tim Rühsen
407cd5f23b Add new fuzzer for the FTP listing parsers
* fuzz/Makefile.am: Add wget_ftpls_fuzzer
* fuzz/wget_ftpls_fuzzer.c: New fuzzer
* fuzz/wget_ftpls_fuzzer.dict: Fuzzer dictionary
* fuzz/wget_ftpls_fuzzer.in/starter: Starting corpus
* src/ftp-ls.c: Parsing function take FILE * as argument,
  new function ftp_parse_ls_fp()
* src/ftp.c: Remove static from freefileinfo()
* src/ftp.h: Add ftp_parse_ls_fp() and freefileinfo()
2018-04-21 19:24:25 +02:00
Tim Rühsen
fbc5f3736e * fuzz/run-clang.sh: Remove -detect_leaks=0 from fuzzer command line 2018-04-21 18:22:37 +02:00
Tim Rühsen
7ecfe3ef70 * src/main.c (main): Fix memleak for fuzzing/testing 2018-04-21 18:21:52 +02:00
Tim Rühsen
7e635d173e * src/init.c: Fix fuzzing in case ~/.wgetrc doesn't exist 2018-04-21 16:33:45 +02:00
Tim Rühsen
e83dd5b0aa Fix fuzzer build for C++
* fuzz/wget_css_fuzzer.c: Include wget.h outside 'extern "C"',
  undef fopen_wgetrc directly after wget.h
* fuzz/wget_html_fuzzer.c: Likewise
2018-04-20 23:59:58 +02:00
Tim Rühsen
0b02993e39 * fuzz/Makefile.am: Add -I/lib to oss-fuzz builds 2018-04-20 23:28:12 +02:00
Tim Rühsen
23b0275feb Add new HTML parser fuzzer
* fuzz/Makefile.am: Add wget_html_fuzzer
* fuzz/wget_html_fuzzer.c: New fuzzer
* fuzz/wget_html_fuzzer.dict: HTML dictionary for fuzzing
* fuzz/wget_html_fuzzer.in: Initial corpora
* src/html-url.c: Add new function get_urls_html_fm()
* src/html-url.h: Add ne function get_urls_html_fm()
* src/wget.h: Fix define for fopen_wgetrc()
2018-04-20 22:33:58 +02:00
Tim Rühsen
77c31d301b * fuzz/wget_css_fuzzer.c: Fix build 2018-04-20 20:06:50 +02:00
Tim Rühsen
30bd99f3f5 * fuzz/wget_css_fuzzer.in/*: Update fuzzer corpora 2018-04-20 20:04:59 +02:00
Tim Rühsen
c9a091ae45 * src/css-url.c (get_uri_string): Fix buffer overflow (read) 2018-04-20 11:37:52 +02:00
Tim Rühsen
7a5db30b01 * src/iri.h: Fix C++ compile error 2018-04-20 10:17:55 +02:00
Tim Rühsen
9d899d7bb7 * src/http.c: Download and scan CSS files in spider mode 2018-04-19 23:05:06 +02:00
Tim Rühsen
d25d036fba * src/css-url.c (get_urls_css): Call yylex_destroy() to reset CSS scanner 2018-04-19 23:05:06 +02:00
Tim Rühsen
c1a368af89 Add new fuzzer wget_css_fuzzer.c
* fuzz/Makefile.am: Add wget_css_fuzzer.c
* fuzz/wget_css_fuzzer.c: New fuzzer
2018-04-19 23:05:06 +02:00
Tim Rühsen
ff3c7733b7 * src/html-url.h: Include needed header files 2018-04-18 20:41:08 +02:00
Tim Rühsen
cc7f15a529 * wget_options_fuzzer.in/*: Update fuzzer corpora 2018-04-18 16:46:44 +02:00
Tim Rühsen
ae6390e6a4 * fuzz/README.md: Add CXXFLAGS and more configure options 2018-04-18 16:43:51 +02:00
Tim Rühsen
3ae58dae13 Fix oss-fuzz issue with exit()
* src/wget.h: Define exit() as exit_wget()
* fuzz/wget_options_fuzzer.c: Implement exit_wget() and cleanup
2018-04-18 13:26:10 +02:00
Tim Rühsen
bb03572c2a * fuzz/wget_options_fuzzer.c: Declare fopen_* as C functions 2018-04-17 23:35:42 +02:00
Tim Rühsen
66b416b6cd Fix fopen/stdin issues with fuzzing
* fuzz/wget_options_fuzzer.c: Add fopen_wget() and fopen_wgetrc()
* src/utils.c: Use fopen_wgetrc() for config files,
  don't read from stdin when fuzzing
* src/wget.h: Define fopen as fopen_wget when fuzzing,
  define fopen_wgetrc as fopen when not fuzzing
2018-04-17 23:02:04 +02:00
Tim Rühsen
fdd86aada9 * configure.ac: AC_DEFINE FUZZING if --enable-fuzzing was given 2018-04-17 15:55:58 +02:00
Tim Rühsen
ce90ed78b6 * fuzz/wget_options_fuzzer.c: Write fuzzer crash reports 2018-04-17 12:41:34 +02:00
Tim Rühsen
3c4a6506a5 * src/log.c: Don't check_redirect_output() when fuzzing 2018-04-17 12:40:47 +02:00
Tim Rühsen
fbb4cd231e * src/main.c (promt_for_password): Avoid getpass() when fuzzing 2018-04-17 12:15:18 +02:00
Tim Rühsen
3ceb6e5630 Fix double fclose() with -d while fuzzing
* src/ftp.c (ftp_loop_internal): Set warc_tmp to NULL after ffclose()
* src/init.c (cleanup): Set output_stream to NULL after fclose()
* src/log.c (log_close): Set global stream vars to NULL after closing
* src/recur.c (retrieve_tree): Set rejectedlog to NULL after closing
* src/warc.c (warc_close): Set stream vars to NULL after closing
2018-04-17 11:59:54 +02:00
Tim Rühsen
eaf167aaaa * src/main.c (main): Don't background if TESTING 2018-04-17 11:50:36 +02:00
Tim Rühsen
7d5de64fc9 * src/init.c (initialize): Return error, don't exit() 2018-04-17 11:42:43 +02:00
Tim Rühsen
70042265be * src/init.c (cmd_use_askpass): Return false on error 2018-04-16 23:04:53 +02:00
Tim Rühsen
64758655c4 * src/utils.c (compile_posix_regex): Hard-code string to regcomp
regcomp() may be too cpu + memory intensive for fuzzing.
See https://sourceware.org/glibc/wiki/Security%20Exceptions
2018-04-16 22:04:54 +02:00
Tim Rühsen
e737c4b10e Fix 2 more memleaks
* src/init.c (initialize): Use global var for wgetrc filename
* src/iri.c (find_locale): Return strdup'ed locale string
* src/options.h (struct options): Add wgetrcfile
2018-04-16 22:02:11 +02:00
Tim Rühsen
05a8c064e9 * src/init.c (cleanup): Set output_stream to NULL after closing 2018-04-16 13:22:29 +02:00
Tim Rühsen
01002a168a Fix homedir memory leaks
* src/hsts.c: Use opt.homedir
* src/init.c: Likewise
* src/main.c: Likewise
* src/netrc.c: Likewise
* src/options.h (struct options): Add homedir
2018-04-16 13:19:03 +02:00
Tim Rühsen
73fd57585c * src/main.c (main): Free opt.encoding_remote properly 2018-04-16 12:21:52 +02:00
Tim Rühsen
7963260e76 * src/host.c (wait_ares): Free ptimer 2018-04-16 11:58:18 +02:00
Tim Rühsen
99a7039def * src/init.c (cleanup): Free regex objects properly 2018-04-16 11:57:39 +02:00
Tim Rühsen
d7e3acb2cc * src/init.c (cleanup): Never call cleanup() twice 2018-04-16 09:58:51 +02:00
Tim Rühsen
e0860dd1ff * src/init.c (cmd_bytes_sum): Fix integer over- and underflow 2018-04-16 09:58:51 +02:00
Tim Rühsen
15ef79f808 * src/main.c (save_hsts): Free hsts_store after closing 2018-04-16 09:58:51 +02:00
Tim Rühsen
79385a29fd Use strtol() instead of selfmade function
* src/init.c (cmd_number): Use strtol() instead of selfmade function
* bootstrap.conf: Add strtol gnulib module
2018-04-16 09:58:51 +02:00
Tim Rühsen
55da9f71f0 * src/hsts.c (hsts_hash_func): Allow integer overflow 2018-04-16 09:58:51 +02:00
Tim Rühsen
bec9816f40 * init.c (cmd_spec_mirror): Fix uninitialzed stack variable 2018-04-16 09:58:51 +02:00
Tim Rühsen
b86294e1c9 * src/init.c (cleanup): Free more variables 2018-04-16 09:58:51 +02:00
Tim Rühsen
7e1e5f3959 * wget_options_fuzzer.in: Add corpora directory 2018-04-16 09:58:51 +02:00
Tim Rühsen
4a6336d551 * fuzz/wget_options_fuzzer.c: Suppress error messages from wget 2018-04-16 09:58:51 +02:00
Tim Rühsen
328438e69b * src/utils.c (fopen_stat): Early return to allow fuzzing/fmemopen 2018-04-16 09:58:51 +02:00
Tim Rühsen
36f029d2f0 * src/init.c (initialize): Free mem before exit() 2018-04-16 09:58:51 +02:00
Tim Rühsen
a4402120ad Add OSS-Fuzz infrastruture
* Makefile.am: Add fuzz/ to SUBDIRS
* cfg.mk: Fix 'make syntax-check'
* configure.ac: Add --enable-fuzzing
* fuzz/Makefile.am: New file
* fuzz/README.md: New file
* fuzz/fuzzer.h: New file
* fuzz/get_all_corpora: New file
* fuzz/get_ossfuzz_corpora: New file
* fuzz/glob_crash.c: New file
* fuzz/main.c: New file
* fuzz/run-afl.sh: New file
* fuzz/run-clang.sh: New file
* fuzz/view-coverage.sh: New file
* fuzz/wget_options_fuzzer.c: New file
* fuzz/wget_options_fuzzer.dict: New file
* src/init.c (cleanup): Free more resources
* src/main.c (init_switches): Initialize only once,
  (print_usage): Don't print if TESTING is defined
* src/utils.h: Include wget.h
2018-04-16 09:58:51 +02:00
Tim Rühsen
de54c970b2 Move unit-test code to tests/
* src/Makefile.am: Remove test.c and test.h
* src/test.c: Rename to tests/unit-tests.c
* src/test.h: Rename to tests/unit-tests.h
* tests/Makefile.am: Add unit-tests.c and unit-tests.h
* src/hsts.c: Amend #include
* src/http.c: Likewise
* src/init.c: Likewise
* src/metalink.c: Likewise
* src/res.c: Likewise
* src/url.c: Likewise
* src/utils.c: Likewise
2018-04-05 15:06:47 +02:00
Tim Rühsen
3e84963e84 * src/main.c: Rename main() -> main_wget() for unit tests 2018-04-05 15:06:47 +02:00
Tim Rühsen
f56f970bc2 Fix some issues found by 'infer' 2018-03-14 14:43:35 +01:00
Tim Rühsen
0b54043d17 * src/openssl.c: Fix build for OpenSSL 1.1.0 without TLS1_3_VERSION 2018-03-08 16:17:14 +01:00
Tim Rühsen
040106b3e2 Add docs for --secure-protocol=TLSv1_3
* doc/wget.texi: Likewise
2018-03-08 15:30:28 +01:00
Loganaden Velvindron
fde8cefd13 Add TLS1.3 support for OpenSSL build
* src/init.c: Add 'tlsv1_3 for --secure-protocol
* src/openssl.c (ssl_init): Enable TLS1.3 if possible
* src/options.h: Add secure_protocol_tlsv1_3
* doc/wget.texi: Add description of TLSv1_3

Copyright-paperwork-exempt: Yes
2018-03-08 15:30:14 +01:00
Tim Rühsen
ba2b0654b4 * src/main.c: Add help text for --retry-on-http-error
Reported-by: Giovanni Tirloni
2018-03-07 10:32:08 +01:00
Tim Rühsen
375bfa98dc * src/url.c (convert_fname): Fix invalid free on iconv_open() failure
Reported-by: Volkmar Klatt
2018-03-01 16:03:29 +01:00
Tim Rühsen
bea54e0da4 * src/mswindows.c: Fix prototype of fork_to_background()
Reported-by: Gisle Vanem
2018-02-21 19:05:15 +01:00
Tim Rühsen
98ec26cc6d Fix warning to not print binary IP address
* tests/Test-https-badcerts.px: Likewise
* tests/Test-https-clientcert.px: Likewise
* tests/Test-https-crl.px: Likewise
* tests/Test-https-pfs.px: Likewise
* tests/Test-https-selfsigned.px: Likewise
* tests/Test-https-tlsv1.px: Likewise
* tests/Test-https-tlsv1x.px: Likewise
* tests/Test-https-weboftrust.px: Likewise
2018-02-09 10:33:07 +01:00
Tim Rühsen
9887b870d1 Use gnulib's utime()
* bootstrap.conf: Add modules utime and utime-h
* src/utils.c (touch): Remove own code for gnulib's utime()
2018-02-09 10:21:43 +01:00
Tim Rühsen
c722973212 Fix logging in background mode
* ../src/main.c: Re-init logfile if changed for background mode
* ../src/utils.c: fork_to_background() returns whether logfile changed
* ../src/utils.h: Set return type bool for fork_to_background()

Fixes: #53020
Reported-by: Noël Köthe
2018-02-09 10:21:43 +01:00
Tim Rühsen
bb7fa977a1 * src/http.c: Fix two typos in comments 2018-02-09 10:18:35 +01:00
Tim Rühsen
f2d6075a6e * testenv/conf/expected_files.py: Ignore pubring.gpg
Reported-by: Arkadiusz Miśkiewicz
2018-01-31 09:04:42 +01:00
Tim Rühsen
0caed28cda * bootstrap.conf: Replace gnulib module gettext by gettext-h 2018-01-31 09:01:54 +01:00
Tim Rühsen
d27032c446 Mention list and bugtracker for --help and in man page
* doc/wget.texi: Mention list and bugtracker in man page
* src/main.c: Mention list and bugtracker for --help
2018-01-22 10:39:49 +01:00
Darshit Shah
080fd0d93b * contrib/make-release: Add another release script 2018-01-21 13:50:45 +01:00
Darshit Shah
ab8a3c80df Update NEWS file for new release 2018-01-21 11:00:29 +01:00
Darshit Shah
11ecabcb3d Fix typo in documentation 2018-01-21 11:00:25 +01:00
Darshit Shah
d0a5d9f131 Switch off compression by default
Gzip compression has a number of bugs which need to be ironed out before
we can support it by default. Some of these stem from a misunderstanding
of the HTTP spec, but a lot of them are also due to many web servers not
being compliant with RFC 7231.

With this commit, I am marking GZip compression support as experimental
in GNU Wget pending further investigation and the addition of tests.

* src/init.c (defaults): Switch of compression support by default
* docs/wget.texi: State that compression is experimental
2018-01-21 10:51:11 +01:00
Darshit Shah
0ff71e5ff1 Pull gnulib module forward 2018-01-21 10:51:09 +01:00
Darshit Shah
0d0a95a01b Revert "* src/init.c (defaults): Set compression_none as the default compression"
This reverts commit 8283ac0846.
2018-01-21 10:50:44 +01:00
Darshit Shah
8283ac0846 * src/init.c (defaults): Set compression_none as the default compression 2018-01-21 10:17:39 +01:00
Reiji
a7cc4e2b37 * src/http.c (gethttp): Fix bug that prevented all files from being decompressed
Signed-off-by: Darshit Shah <darnir@gnu.org>
2018-01-20 14:04:28 +01:00
Tim Rühsen
55d25fc20c * src/host.c (sufmatch): Fix to domain matching 2018-01-19 19:32:01 +01:00
Gisle Vanem
513cc1c0c8 * src/netrc.c: Fix Standalone compilation of netrc file 2018-01-17 14:44:52 +01:00
Darshit Shah
3249045a91 * NEWS: Update NEWS file for new release 2018-01-14 12:17:31 +01:00
Darshit Shah
b1fc37f465 * bootstrap: Use the faster gnulib-tool.py script if possible 2018-01-14 11:33:52 +01:00
Darshit Shah
953bcdaba8 Sync the bootstrap script from gnulib 2018-01-14 11:28:51 +01:00
Darshit Shah
183fccdaad Update Copyright years 2018-01-14 11:24:43 +01:00
Darshit Shah
8240439e21 Replace HTTP urls with HTTPS where valid 2018-01-14 11:15:32 +01:00
Darshit Shah
a03716a0fe Update gnulib 2018-01-14 11:15:16 +01:00
Darshit Shah
d26c6c0028 * src/netrc.c: Search for the correct netrc file on Windows 2018-01-14 10:55:03 +01:00
Tim Rühsen
047746eb76 * src/http.c: Exclude *.gz and *.tgz from decompression 2018-01-10 15:46:13 +01:00
Tim Rühsen
d8df356d4b * src/utils.c (wg_pin_peer_pubkey): Fix format warning 2017-12-31 13:03:25 +01:00
Peter Wu
220c24ecb5 Avoid redirecting output to file when tcgetpgrp fails
* src/log.c (check_redirect_output): tcgetpgrp can return -1 (ENOTTY),
be sure to check whether a valid controlling terminal exists before
redirecting.

Fixes: #51181
2017-12-31 12:59:15 +01:00
Darshit Shah
3d2b2231cd Add new test for 416 responses
* testenv/server/http/http_server.py: If there are multiple requests in
which the requested range is unsatisfiable, then send a body in the in
the 2nd response onwards
* testenv/Test-416.py: New test to check how Wget handles 416 responses
2017-12-11 14:48:01 +01:00
Darshit Shah
693cee0109 Don't assume a 416 response has no body
* http.c(gethttp): In case of a 416 response, try to drain the socket of
any bytes before reusing the connection

Reported-By: Iru Cai <mytbk920423@gmail.com>
2017-12-08 18:44:17 +01:00
Tim Rühsen
6aa6b669ef Support building with OpenSSL 1.1 w/o deprecated features
* src/openssl.c (ssl_init): Fix code for the subject's issue

Reported-by: Matthew Thode
2017-11-26 18:59:47 +01:00
Tim Rühsen
8551ceccfe Avoid link conversion after 304 Not Modified
* src/http.c (gethttp): Handle 304 before setting document content type

Fixes: #52404
Reported-by: Ben Fuchs
2017-11-25 19:33:03 +01:00
YX Hao
19060db44f Fix printing mutibyte chars as unprintable chars on Windows
* src/log.c (get_warc_log_fp): Fix return value to stderr
* src/main.c (main): Init logging as soon as possible,
  fix locale/charset on Windows
2017-11-16 12:23:20 +01:00
Tim Rühsen
a30a5a9059 * tests/Test-https-weboftrust.px: Fix/add sslport 2017-11-16 12:12:16 +01:00
YX Hao
a9a953feee Convert remote path to local encoding
* src/url.c (url_file_name): Convert remote path to local encoding
2017-11-15 19:58:53 +01:00
Tim Rühsen
267cd51fff Do not use must-revalidate in Cache-Control header
As the bug report states, 'must-revalidate' is a request directive.

Fixes #52379
2017-11-10 10:57:46 +01:00
Darshit Shah
973c26ed7d Fix Segfault due to derefencing null ptr
* src/http.c(gethttp): When Encoding is gzip, ensure that the
Content-Type Header was actually seen. Without this, the "type" variable
is null causing a Segfault.

Reported-By: Noël Köthe <noel@debian.org>
2017-11-06 10:09:03 +01:00
Tim Rühsen
16d066f89c * src/http.c: Fix H_REDIRECTED 2017-11-03 22:23:04 +01:00
Tim Rühsen
a2477d487c * src/http.c: Add support for HTTP status code 308 2017-11-03 22:12:11 +01:00
Tim Rühsen
7ac72e063c Fix uninitialized value messages in tests
* tests/Test-https-*.px: Use correct variable in error message

Reported-by: Jeffrey Walton
2017-10-30 21:29:14 +01:00
Tim Rühsen
95891cbba3 Ignore dirmngr.conf, gpg.conf in test directories
* testenv/conf/expected_files.py: Ignore dirmngr.conf, gpg.conf

Libgpgme creates certain files if $HOME doesn't contain the .gnupg directory.
These files disturb some metalink tests if we don't ignore them.

Reported-by: Arkadiusz Miśkiewicz <arekm@maven.pl>
2017-10-30 13:04:09 +01:00
Tim Rühsen
2486320a2a * contrib/release: Cleanup, use lzip instead of xz 2017-10-27 11:26:37 +02:00
Tim Rühsen
ba6b44f674 Fix heap overflow in HTTP protocol handling (CVE-2017-13090)
* src/retr.c (fd_read_body): Stop processing on negative chunk size

Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
2017-10-26 17:29:38 +02:00
Tim Rühsen
d892291fb8 Fix stack overflow in HTTP protocol handling (CVE-2017-13089)
* src/http.c (skip_short_body): Return error on negative chunk size

Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
2017-10-26 17:29:38 +02:00
Tim Rühsen
bec4c215a8 Update for release 1.19.2
* NEWS: Add news for 1.19.2
* doc/wget.texi: Fix 1.20 to 1.19.2
2017-10-26 16:31:57 +02:00
YX Hao
27d78d944f Avoid unnecessary UTF-8 encoded fallback (trivial change)
* src/retr.c (retrieve_url): Check for changed URL on redirect
2017-10-25 14:26:36 +02:00
Tim Rühsen
60f033426f Add GNU extensions to .netrc parsing
src/netrc.c (parse_netrc): Add 'port' and 'force' extensions

Reported-by: September 20Tim Landscheidt
2017-09-27 12:42:06 +02:00
Josef Moellers
6f3b995993 Bail out on unexpected 416 server errors
* src/http.c (gethttp): Stop on 416 if file is incomplete
2017-09-18 16:45:49 +02:00
Tim Schlueter
c451eec155 Add gzip Content-Encoding decompression
* src/http.c (struct http_stat): Add remote_encoding field.
(read_response_body): Enable gzip decompression.
(initialize_request): Send gzip Accept-Encoding header.
(gethttp): Decompress files with gzip Content-Encoding.
* src/retr.c: include zlib.h.
(zalloc): New function.
(zfree): New function.
(fd_read_body): Decompress gzip data.
* src/retr.h (fd_read_body enum): Add rb_compressed_gzip flag.
2017-08-04 14:34:53 +02:00
Tim Schlueter
b543dfe783 Add --compression option
* doc/wget.texi: Add --compression documentation.
* src/init.c (cmd_spec_compression): New function.
(commands[]): Add opt.compression.
(defaults): Set default opt.compression value.
* src/main.c (option_data[]): Add struct for --compression.
(print_help, help[]): Add description for --compression.
(main): Add incompatibility checks for --compression.
* src/options.h (struct options): Add compression enum and field.
2017-08-04 14:34:53 +02:00
Tim Schlueter
08ed2a5530 Adjust Extension based on Content-Encoding
* doc/wget.texi (--adjust-extension, adjust_extension): Updated documentation.
* src/http.c (encoding_t): New enum.
(struct http_stat): Add local_encoding field.
(gethttp): --adjust-extension based on Content-Encoding.
2017-08-04 14:34:53 +02:00
Darshit Shah
951d3e4cdd Document gperf as a requirement 2017-07-31 14:56:32 +02:00
Tim Rühsen
3ad3b3e36c * src/url.c (url_scheme): Use ASCII version of strncasecmp 2017-07-28 17:11:26 +02:00
Tim Rühsen
5fb6b6bd68 Fix misuse of strncasecmp
* src/http.c (set_content_type): Use c_strcasecmp instead of strncasecmp

See issue bug #51576
2017-07-28 16:56:27 +02:00
Tim Rühsen
f42229b1fd Fix python test suite for GnuTLS 3.5.12+
* testenv/Test-*.py: Replace 127.0.0.1 by localhost
* testenv/certs/server-template.cfg: Likewise
* testenv/certs/server-cert.pem: Regenerate
* testenv/certs/server-crl.pem: Likewise
* testenv/test/base_test.py: Hardcode 'localhost' as server domain

Reported-by: Ludovic Courtès <ludo@gnu.org>
2017-07-09 11:39:05 +02:00
Tim Rühsen
21154bdc36 Check for 304 response before applying --adjust-extension
* src/http.c (gethttp): Move 304 code before --adjust-extension code

This fixes applying --adjust-extension in combination with 304
HTTP responses. It could lead to .html extensions to arbitrary
files.

Reported-by: anfractuosity
2017-06-13 11:25:20 +02:00
Tim Rühsen
ae293c945a Fix buffer overflow in Public Key Pinning
* src/utils.c (wget_base64_decode): Add param for destination size,
  (wg_pubkey_pem_to_der): Amend call to wget_base64_decode(),
  (wg_pin_peer_pubkey): Likewise and fix code style.
* src/utils.h: Add param to wget_base64_decode()
* src/http-ntlm.c (ntlm_input): Amend call to wget_base64_decode()
* src/http.c (skip_content_type): Likewise

Fixes #51227
2017-06-13 10:23:04 +02:00
Tim Rühsen
407c1f990a * doc/wget.texi: Mention --no-config 2017-06-02 10:13:22 +02:00
Tim Rühsen
86b46a34a5 * testenv/Test-recursive-basic.py: Check crawled files 2017-05-16 11:20:57 +02:00
Tim Rühsen
5d4ada1b7b Fix two Metalink tests if $HOME is changed
* conf/expected_files.py (gen_local_fs_snapshot): Skip processing
  of 'pubring.kbx'
2017-05-16 10:24:52 +02:00
Tomas Hozza
876def8ebe Add command line option to disable use of .netrc
Although internally code uses option for (not) reading .netrc for
credentials, it was not possible to turn this behavior off on command
line. Note that it was possible to turn it off using wgetrc.

Idea for this change came from Bruce Jerrick (bmj001@gmail.com).
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1425097

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tomas Hozza
f8c3df1f40 Fixed getting of credentials from .netrc
There seemed to be a copy&paste error in http.c code, which decides
whether to get credentials from .netrc. In ftp.c "user" and "pass"
variables are char*, while in http.c, these are char**. For this reason
they should be dereferenced when determining if password and user login
is set to some value.

Also since both variables are dereferenced on lines above the changed
code, it does not really make sense to check if they are NULL.

This patch is based on fix from Bruce Jerrick <bmj001@gmail.com>.
Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425097

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tomas Hozza
17960b57d5 Added tests for HTTP authentication using credentials from .netrc
Getting credentials from .netrc has been broken from time to time, thus
adding a test coverage to prevent regressions.

Also added setting of "HOME" environment variable when executing wget,
to make sure LocalFiles like .netrc, which are created just for the
test, are actually used.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-15 16:06:50 +02:00
Tim Rühsen
40c0d30f5c Fix Test-https-badcerts.px
* util/createcerts.sh: Set date of invalid cert to +20 years
* tests/certs/invalid.crt: Re-generated
* tests/certs/invalid.key: Re-generated
2017-05-15 15:46:11 +02:00
Tim Rühsen
e2c702571f * util/createcerts.sh: Fix double equal (syntax-check) 2017-05-14 21:35:34 +02:00
Tim Rühsen
269fb6201d Fix Test-https-badcerts.px to work with GnuTLS
* tests/Test-https-badcerts.px: Remove OpenSSL requirement
* tests/certs/expired.crt: Re-created with valid dates
* tests/certs/expired.key: Likewise
* tests/certs/invalid.crt: Likewise
* tests/certs/invalid.key: Likewise
2017-05-14 11:10:15 +02:00
Vijo Cherian
c08778aeb3 New shell script to create the certs and keys required for TLS tests
* util/createcerts.sh: New file
2017-05-14 11:07:43 +02:00
Tim Rühsen
297c1e2ac3 Fix HTTPS testing for stricter OpenSSL
* testenv/certs/README: Amend cert creation extensions
* testenv/certs/ca-cert.pem: Created without OCSP signing purpose

Having the OCSP signing purpose set made newer versions of OpenSSL
fail due to stricter checking. Test version of OpenSSL was 1.1.0e.
2017-05-11 12:38:19 +02:00
Tim Rühsen
936efc3564 * src/iri.c (idn_encode): Better IDNA 2003 compatibility 2017-05-11 11:56:26 +02:00
Tim Rühsen
11d3de74ca * .gitlab-ci.yml: Also test OpenSSL build 2017-05-10 12:12:42 +02:00
Tim Rühsen
35d5b67cc1 * .gitlab-ci.yml: Remove installation of texlive 2017-05-10 10:41:42 +02:00
Tim Rühsen
da50a1ecc3 Add certs/wotca.pem to avoid temp. file creation
* tests/certs/wotca.pem: New file
* tests/Test-https-weboftrust.px: Remove creation of wotca.pem

This change let Test-https-weboftrust.px survive VPATH builds.
2017-05-10 10:34:42 +02:00
Tim Rühsen
1068ca07d3 * cfg.mk: Exclude ^tests/certs/.* from syntax-check 2017-05-10 10:05:18 +02:00
Tim Rühsen
0666e0f375 * po/POTFILES.in: Remove spider.c (syntax-check) 2017-05-10 10:04:34 +02:00
Tim Rühsen
0d9f8280e0 * tests/Test-https-selfsigned.px: Add newline at EOF (syntax-check) 2017-05-10 10:04:30 +02:00
Vijo Cherian
17f0e16ecc Removed all uses of temp files.
Added needed files to GIT
2017-05-09 16:45:13 -07:00
Vijo Cherian
cc74f8b7e2 Made CRL related files in the repo, instead of trying to generate them 2017-05-09 15:37:28 -07:00
Tim Rühsen
2894a37189 Cleanup on exit in Test-https-*.px
* tests/Test-https-*.px: Cleanup on exit
2017-05-09 16:58:33 +02:00
Tim Rühsen
4669ba5e0c Auto-generate interca.conf and rootca.conf
* configure.ac: Add interca.conf.in and rootca.conf.in to AC_CONFIG_FILES
* tests/certs/interca.conf: Removed
* tests/certs/rootca.conf: Removed
* tests/certs/interca.conf.in: New file
* tests/certs/rootca.conf.in: New file
2017-05-09 16:04:48 +02:00
Tim Rühsen
b5c2d083ea Fix path and VPATH issues of new https/TLS tests
* tests/SSLTest.pm: Use $srcdir in read-only files,
  (_setup_server): Remove unneeded path fixation code
* tests/Test-https-*.px: Use $srcdir for read-only files,
  use $cdir for writable files
* tests/certs/interca.conf: Adjust paths
* tests/certs/rootca.conf: Adjust paths
2017-05-09 15:21:23 +02:00
Tim Rühsen
466afc62b0 Check for test server name resolution in tests
* tests/Test-https-*.px: Skip if test server name resolution fails
2017-05-09 11:16:30 +02:00
Tim Rühsen
f68d001626 * tests/Makefile.am: Enable Test-https-badcerts again 2017-05-09 11:16:30 +02:00
Tim Rühsen
ffe75d0867 Fix WgetFeature.pm to allow multiple required features
* tests/WgetFeature.cfg: Remove file
* tests/WgetFeature.pm: Extend to multiple features, cleanup
2017-05-09 11:16:30 +02:00
Tim Rühsen
9aa894853f * .gitlab-ci.yml: Add wgettestingserver to /etc/hosts 2017-05-09 11:08:32 +02:00
Tim Rühsen
5337b94ce5 * tests/SSLServer.pm: Check for IO::Socket::SSL 2017-05-09 11:08:32 +02:00
Tim Rühsen
a26e6f3527 * tests/Test-https-*: Change server port to <= 32767 2017-05-09 11:08:32 +02:00
Tim Rühsen
b9fb74ddfa Move https test server ports from >32767 to <= 32767
* Test-https-badcerts.px: Change port
* Test-https-crl.px: Likewise
* Test-https-weboftrust.px: Likewise
2017-05-09 11:08:32 +02:00
Tim Rühsen
3132049ae4 * tests/Makefile.am: Add SSLTest.pm and SSLServer.pm to EXTRA_DIST 2017-05-09 11:08:32 +02:00
Tim Rühsen
c0c42da653 * tests//Makefile.am: Disable Test-https-badcerts.px 2017-05-09 11:08:32 +02:00
Tim Rühsen
888cc82c9d Add Gitlab CI (Debian)
* .gitlab-ci.yml: New file
2017-05-09 11:08:32 +02:00
Tim Rühsen
fce1b689e8 * .travis.yml: Use trusty for libidn2-dev 2017-05-06 13:03:28 +02:00
Tim Rühsen
580067d1e6 * tests/certs/test-ca-key.pem: Add newline at EOF 2017-05-04 16:51:49 +02:00
Tim Rühsen
5c4cc011fe Add static HOSTSALIAS file
* tests/certs/wgethosts: New file
* tests/Test-https-*.px: Remove creation of wgethosts file
2017-05-04 16:51:49 +02:00
Vijo Cherian
2a96249469 Added new tests for SSL
* tests/Test-https-badcerts.px : New file
* tests/Test-https-clientcert.px : New file
* tests/Test-https-crl.px : New file
* tests/Test-https-weboftrust.px : New file
* tests/certs/interca.conf : New file
* tests/certs/rootca.conf : New file
* tests/certs/test-ca-key.pem : New file

Added all new SSL / HTTPS tests to make check
Added Test for SSL Web of Trust, accept only if CA chain of trust is intact.
Added a test script for client certificate
Added Test for crlfile option of wget
Added test to make sure that wget doesn't accept expired or invalid certs

Some clean up : 1, Removed cause of warnings from perl & other cosmetic changes
                2, Fix  make -j 4 check such that it passes all tests
2017-05-04 16:51:49 +02:00
Tomas Hozza
0b41c7543a Mention TLSv1_1 and TLSv1_2 as secure-protocol values in help
* src/main.c: The --secure-protocol option accepts also values TLSv1_1
and TLSv1_2, as mentioned in the man page. However the help message
doesn't mention these two values. This patch adds TLSv1_1 and TLSv1_2 as
possible values to the help message.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2017-05-04 14:51:54 +02:00
Tim Rühsen
c4a2b2e77e * src/http.c (gethttp): Support Wayback Machine's X-Archive-Orig-last-modified 2017-05-03 16:37:11 +02:00
Vijo Cherian
f6376ac0dc Added new tests for SSL
* tests/Test-https-badcerts.px : New file
* tests/Test-https-clientcert.px : New file
* tests/Test-https-crl.px : New file
* tests/Test-https-weboftrust.px : New file
* tests/certs/interca.conf : New file
* tests/certs/rootca.conf : New file
* tests/certs/test-ca-key.pem : New file

Added all new SSL / HTTPS tests to make check
Added Test for SSL Web of Trust, accept only if CA chain of trust is intact.
Added a test script for client certificate
Added Test for crlfile option of wget
Added test to make sure that wget doesn't accept expired or invalid certs

Some clean up : Removed cause of warnings from perl & other cosmetic changes
2017-04-28 12:22:54 +02:00
Vijo Cherian
ac519c041e Added new tests for SSL
* tests/SSLServer.pm: New file
* tests/SSLTest.pm: New file
* tests/Test-https-pfs.px: New file
* tests/Test-https-selfsigned.px: New file
* tests/Test-https-tlsv1.px: New file
* tests/Test-https-tlsv1x.px: New file
* tests/certs/server.crt: New file
* tests/certs/server.key: New file
* tests/certs/test-ca-cert.pem: New file

Added 4 new test scripts all for SSL.
Added base pm for SSL testing.
Added SSL tests for TLSv1, TLSv1_1 and PFS.
Added test for self signed cert : check that it fails without
--no-check-certificate and passes with that flag.
2017-04-19 12:59:27 +02:00
Tim Rühsen
56c78c4b09 * src/utils.c: Remove non-portable __builtin_unreachable() 2017-04-18 13:22:25 +02:00
Tim Rühsen
0ec46cb109 Skip iconv() usage if HAVE_ICONV is not defined
This helps on broken iconv implementations, e.g. Solaris.

Reported-by: Mojca Miklavec
2017-04-18 13:17:19 +02:00
Tim Rühsen
67cb37ce5a Mention 'bash' for executing ./bootstrap
Reported-by: Mojca Miklavec
2017-04-18 10:47:15 +02:00
Tim Rühsen
0004d3ec22 * bootstrap.conf: Make 'sed' options more portable
'sed -i' works differently on FreeBSD.

Reported-by: Mojca Miklavec
2017-04-18 10:44:06 +02:00
Tim Rühsen
92bfe2a2e4 Fix charset transcoding issue for non-reversible codepoints
* src/url.c: Check iconv() against 0, not -1

On some libiconv implementations, unknown codepoints become
encoded as ?, e.g. when converting a non-ascii codepoint to ASCII.
This results in ambigious file names which also fails our tests.
2017-04-16 19:55:14 +02:00
Tim Rühsen
fc2f4233ed * src/iri.c: Fix WIN32 idn2_free, forgotten code 2017-04-16 19:50:10 +02:00
Darshit Shah
b2c38d33e1 * src/init.c: Set flstats correctly when using WGETRC env var 2017-04-14 01:16:49 +02:00
Tim Rühsen
6ef493b19e Fix use of idn2_free()
* src/connect.c (connect_to_ip): Use xfree() instead of idn2_free()
* src/host.c (lookup_host): Use xfree() instead of idn2_free()
* src/iri.h: Do not include idn2.h
* src/url.c (url_free): Use xfree() instead of idn2_free()
* src/url.h (struct url): Remove 'idn_allocated' from struct

Reported-by: Gisle Vanem
2017-04-08 11:05:55 +02:00
Anton Yuzhaninov
7ffe93cabb Fix perl warnings in tests
* tests/FTPServer.pm: Escape '{' in RE to fix warnings
* tests/FTPTest.pm: Likewise
* tests/HTTPServer.pm: Likewise
* tests/HTTPTest.pm: Likewise
* tests/Test-proxied-https-auth-keepalive.px: Likewise
* tests/Test-proxied-https-auth.px: Likewise
Escape '{' in RE to fix warnings:
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/{{ <-- HERE port}}/
2017-04-05 20:04:12 +02:00
klemens
f381831d88 Fix typos in comments 2017-04-01 19:38:09 +02:00
Tim Rühsen
02d40a4676 * src/metalink.c (retrieve_from_metalink): Fix len in memset() 2017-03-31 13:15:27 +02:00
Tim Rühsen
e89267fbbc Add gnulib module group-member 2017-03-24 11:36:01 +01:00
Vijo Cherian
400b8eba6c Safeguards against TOCTTOU
* src/utils.h: Add struct file_stat_s declaration,
  change prototypes of file_exists_p(),
  add prototypes for fopen_stat() and open_stat().
* src/utils.c: Extend file_exists_p(),
  new function fopen_stat() and open_stat(),
  add new param for file_exists_p().
* src/init.h: Add param file_stats_t to run_wgetrc().
* src/ftp.c: Amend calls to extended functions.
* src/hsts.c: Likewise.
* src/http.c: Likewise.
* src/init.c: Likewise.
* src/main.c: Likewise.
* src/metalink.c: Likewise.
* src/retr.c: Likewise.
* src/url.c: Likewise.

Added fopen_stat() and open_stat() that checks to makes sure the file didn't
change underneath us.
Return error from file_exists_p().
Added a way to return error from this file without major surgery to the
callers.

Fixes: #20369
2017-03-24 09:39:09 +01:00
Tim Rühsen
90a0a7499c Update gnulib 2017-03-24 09:19:52 +01:00
Christof Horschitz
1d71645c06 * src/warc.c (warc_write_cdx_record): Escape URLs 2017-03-22 15:01:04 +01:00
Mike Frysinger
e249844143 Include libunistring headers only when used
* src/iri.c: Check for libidn2 < 0.14 to include libunistring headers

The unistring functions are used only when an older version of libidn2
is used, so don't include its headers either w/newer libdin2 versions.
2017-03-20 09:39:20 +01:00
Tim Rühsen
84a93f4127 Fix links to www.robotstxt.org
* NEWS: Fix links
* doc/wget.texi: Likewise
* src/res.c: Likewise

Reported-by: Noël Köthe
2017-03-18 19:05:38 +01:00
Tim Rühsen
f31b93424b * tests/WgetTests.pm: Add -d to Wget test options 2017-03-13 21:31:09 +01:00
Tim Rühsen
90b487369a Include <arpa/inet.h> for Windows
Reported-by: Gisle Vanem
2017-03-08 13:00:54 +01:00
Tim Rühsen
57d748117f Fix updating HSTS entries
* src/hsts.c (hsts_store_entry): Always update 'created' field

Fixes: #50490
Reported-by: Deian Stefan, Atyansh Jaiswal, Jonathan Luck
2017-03-08 10:56:12 +01:00
Tim Rühsen
4d729e322f Fix CRLF injection in Wget host part
* src/url.c (url_parse): Reject control characters in host part of URL

Reported-by: Orange Tsai
2017-03-06 10:04:22 +01:00
Benjamin Esham
63c2aea255 * src/warc.c: Use warc_write_header_uri for all WARC-Target-URI fields
The WARC spec requires that all URIs be enclosed in angle brackets. This
was being done in most cases, but not for "WARC-Target-URI" fields in
WARC blocks of type "response", "resource", "revisit", and "metadata".
2017-03-04 12:51:39 +01:00
Tim Rühsen
ac4fed3220 Fix 504 status handling
* src/http.c (gethttp): Move 504 handling to correct place.
  (http_loop): Fix memeory leak.
* testenv/server/http/http_server.py: Add Content-Length header on non-2xx
  status codes with a body

Reported-by: Adam Sampson
2017-02-16 15:53:56 +01:00
YX Hao
cf5df5593d * src/url.c (url_file_name): Do not charset convert local directory
In a non-ASCII environment, the local path may contain non-ASCII
characters. The server responded file name must be converted before
it is concatenated to the local path. Conversion after concatenation
may result in 'iconv' errors.
2017-02-16 12:52:16 +01:00
Tim Rühsen
2215ee8d10 * configure.ac: Remove manually resetting of LIBICONV variable
Fixes: #48193
Reported-by: ilovezfs
Reported-by: Charles
2017-02-15 20:03:58 +01:00
Tim Rühsen
32e26dc199 * bootstrap.conf: Call gperf to create lib/unicase/special-casing-table.h 2017-02-15 14:45:23 +01:00
Tim Rühsen
21ac4ae356 * bootstrap.conf: Fix latest gnulib to work with gperf < 3.1 2017-02-15 13:03:10 +01:00
Darshit Shah
838c185ca9 Pull GNULib to latest.
* gnulib: Pull to latest commit. This fixes an issue with Gperf 3.1
2017-02-14 15:08:50 +01:00
Tim Rühsen
ac9be9b756 * src/main.c: Remove double 'verbose' option
Fixes: #50290
2017-02-12 21:18:23 +01:00
Tim Rühsen
6620c5156f * NEWS: update 2017-02-11 12:03:54 +01:00
Tim Rühsen
aebd49d9d4 isrc/http.c (check_retry_on_http_error): Fix gcc warning 2017-02-11 11:54:21 +01:00
Tom Szilagyi
d6d00006a0 Add support for --retry-on-http-error
* doc/wget.text: Add documentation
* src/http.c: Add function check_retry_on_http_error ()
* src/init.c: Add opt.retry_on_http_error
* src/main.c: Add struct for retry-on-http-error to option_data[]
* src/options.h: Add retry_on_http_error to struct options
2017-02-11 11:50:24 +01:00
Tim Rühsen
05acf5d3f6 Revert "Add support for --retry-on-http-error"
This reverts commit 977276374d.
2017-02-11 11:45:11 +01:00
Tim Rühsen
80c62c238e Change libtool library deps to non-libtool deps
Reported-by: Yousong Zhou
Fixes: #50260
2017-02-10 17:20:42 +01:00
Tom Szilagyi
977276374d Add support for --retry-on-http-error
* doc/wget.texi: Add description for --retry-on-http-error
* src/http.c (gethttp):
Consider given HTTP response codes as non-fatal, transient errors.
Supply a comma-separated list of 3-digit HTTP response codes as
argument. Useful to work around special circumstances where retries
are required, but the server responds with an error code normally not
retried by Wget. Such errors might be 503 (Service Unavailable) and
429 (Too Many Requests). Retries enabled by this option are performed
subject to the normal retry timing and retry count limitations of
Wget.

Using this option is intended to support special use cases only and is
generally not recommended, as it can force retries even in cases where
the server is actually trying to decrease its load. Please use it
wisely and only if you know what you are doing.

Example use and a starting point for manual testing:
  wget --retry-on-http-error=429,503 http://httpbin.org/status/503
2017-02-09 21:17:20 +01:00
Tim Rühsen
785c4b92dd * tests/WgetTests.pm: Add --no-config to wget invocation
Without --no-config, wgetrc commands may disturb the tests.

Reported-by: Dagobert Michelsen
2017-02-09 16:34:27 +01:00
Tim Rühsen
d7c34dc0a8 * tests/WgetTests.pm (_verify_download): Fix failure message
Reported-by: Zhiming Wang
2017-02-06 23:01:08 +01:00
Tim Rühsen
d061e553a1 * src/http.c (initialize_request): Fix regression in .netrc auth
Reported-by: Axel Reinhold
2017-02-06 21:44:18 +01:00
Tim Rühsen
2ddd2b69e4 * src/iri.c (idn_encode): Fix memory leak 2017-02-06 21:39:44 +01:00
Tim Rühsen
990f3d67d3 Remove skipping libunistring with --disable-iri
* configure.ac: Remove checking for IRI=no

On certain systems, gnulib falls back to libunistring
for mbtowc() and wcwidth() (used in src/progress.c).

Reported-by: Zhiming Wang
2017-02-06 11:25:48 +01:00
Tim Rühsen
d680b0b551 * bootstrap.conf: Add gnulib module wcwidth 2017-02-04 19:30:38 +01:00
Tim Rühsen
31ac36e170 Fix include/define clash with gnulib's unlink module
* src/options.h: Rename options.unlink to options.unlink_requested
* src/init.c: Replace options unlink member by unlink_requested
* src/http.c: Likewise
* src/ftp.c: Likewise
2017-02-04 18:02:54 +01:00
Tim Rühsen
51b401f5c4 * bootstrap.conf: Specify unicode modules more exact 2017-02-04 18:01:44 +01:00
Tim Rühsen
f2c4289557 * src/xattr.h: Fix #define fsetxattr for MacOS and FreeBSD
Reported-by: Zhiming Wang
2017-02-04 15:29:44 +01:00
Tim Rühsen
58546e52e2 * contrib/release: New release helper script 2017-02-03 16:16:30 +01:00
Tim Rühsen
820f3fc2b4 * cfg.mk: Fix syntax-check 2017-02-03 16:16:30 +01:00
Tim Rühsen
cc8464206d * doc/announcement_1.19.txt: Add new file 2017-02-03 15:19:05 +01:00
9570 changed files with 39735 additions and 1696 deletions

189
.gitlab-ci.yml Normal file
View File

@@ -0,0 +1,189 @@
# we utilize the images generated by the build-images project, to
# speed up CI runs. We also use ccache and store config.cache
# to speed up compilation. We include a version number in cache
# name to allow expiration of old caches.
cache:
key: "$CI_JOB_NAME-ver5"
paths:
- cache/
before_script:
# CCache Config
- mkdir -p cache
- export CCACHE_BASEDIR=${PWD}
- export CCACHE_DIR=${PWD}/cache
- echo $CCACHE_DIR
- export CC="ccache gcc"
after_script:
# somehow after_script loses environment
- export CCACHE_BASEDIR=${PWD}
- export CCACHE_DIR=${PWD}/cache
- ccache -s
variables:
BUILD_IMAGES_PROJECT: gnuwget/build-images
DEBIAN_BUILD: buildenv-debian-stretch
FEDORA_BUILD: buildenv-f27
CENTOS7_BUILD: buildenv-centos7
MINGW_BUILD: buildenv-mingw
ALPINE_BUILD: buildenv-alpine
ARCH_BUILD: buildenv-arch
BASIC_BUILD: buildenv-basic
GET_SOURCES_ATTEMPTS: "3"
GIT_DEPTH: "5"
CONFIGURE_BASE_FLAGS: --enable-assert --cache-file cache/config.cache
CFLAGS_DEFAULT: -O0 -g -ggdb3 -Wall -Wextra
VPATH/Debian:
image: $CI_REGISTRY/$BUILD_IMAGES_PROJECT:$DEBIAN_BUILD
script:
- cp -a /builds/common/gnulib .
- export CFLAGS=$CFLAGS_DEFAULT
- ./bootstrap --skip-po
- mkdir vpath && cd vpath
- ../configure $CONFIGURE_BASE_FLAGS --cache-file ../cache/config.cache
- make -j$(nproc)
- make -j$(nproc) distcheck
tags:
- shared
- docker
artifacts:
expire_in: 2 weeks
when: on_failure
paths:
- ./*.log
- fuzz/*.log
- tests/*.log
- testenv/*.log
# In this build we combine
# * syntax-check
# * abi-check
# * build/valgrind-check
# * build/asan-check
# * build w/Werror and ubsan-check
OpenSSL/Debian:
image: $CI_REGISTRY/$BUILD_IMAGES_PROJECT:$DEBIAN_BUILD
script:
- echo "127.0.0.1 wgettestingserver" >>/etc/hosts
- cat /etc/hosts
- alias make="make -j$(nproc)"
- ./bootstrap --skip-po
- ./configure $CONFIGURE_BASE_FLAGS --with-ssl=gnutls
- make syntax-check
- make check
- make distcheck
- make check-valgrind
tags:
- shared
artifacts:
expire_in: 2 weeks
when: on_failure
paths:
- ./*.log
- fuzz/*.log
- tests/*.log
- testenv/*.log
GnuTLS/Debian:
image: $CI_REGISTRY/$BUILD_IMAGES_PROJECT:$DEBIAN_BUILD
script:
- echo "127.0.0.1 wgettestingserver" >>/etc/hosts
- cat /etc/hosts
- alias make="make -j$(nproc)"
- ./bootstrap --skip-po
- ./configure $CONFIGURE_BASE_FLAGS --with-ssl=openssl
- make syntax-check
- make check
- make distcheck
- make check-valgrind
tags:
- shared
artifacts:
expire_in: 2 weeks
when: on_failure
paths:
- ./*.log
- fuzz/*.log
- tests/*.log
- testenv/*.log
#Centos7 build:
# image: $CI_REGISTRY/$BUILD_IMAGES_PROJECT:$CENTOS7_BUILD
# script:
# - sed -i 's/AM_GNU_GETTEXT_VERSION(0.19.3)/AM_GNU_GETTEXT_VERSION(0.18.2)/g' configure.ac
# - ./bootstrap && ./configure --enable-gcc-warnings --disable-doc && make -j$(nproc) check
# tags:
# - shared
# artifacts:
# expire_in: 2 weeks
# when: on_failure
# paths:
# - tests/*.log
# - compat_reports/
#clang-analyzer/Fedora:
# image: $CI_REGISTRY/$BUILD_IMAGES_PROJECT:$FEDORA_BUILD
# script:
# - ./bootstrap
# - scan-build ./configure --disable-doc --enable-gcc-warnings
# - make -j$(nproc) -C gl
# - scan-build --status-bugs -o scan-build-lib make -j$(nproc) -C lib
# tags:
# - shared
# except:
# - tags
# artifacts:
# expire_in: 2 weeks
# when: on_failure
# paths:
# - scan-build-lib/*
#pages:
# image: $CI_REGISTRY/$BUILD_IMAGES_PROJECT:$DEBIAN_BUILD
# script:
# - ./bootstrap
# - CFLAGS="-g -O0" ./configure --enable-gtk-doc --enable-code-coverage --disable-gcc-warnings
# - make -j$(nproc) check
# - make local-code-coverage-output
# - mkdir -p public
# - rm -rf public/coverage
# - mv libidn2-*-coverage public/coverage
# - rm -rf public/reference
# - mv doc/reference/html public/reference
# - make -C doc manual
# - rm -rf public/manual
# - mv doc/manual public/manual
# tags:
# - shared
# artifacts:
# when: on_success
# paths:
# - public
# only:
# - master
#MinGW64:
# image: $CI_REGISTRY/$BUILD_IMAGES_PROJECT:$FEDORA_BUILD
# before_script:
# - mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc &&
# echo ':DOSWin:M::MZ::/usr/bin/wine64:' > /proc/sys/fs/binfmt_misc/register
# script:
# - ./bootstrap
# - mingw64-configure --disable-valgrind-tests --disable-doc
# - mingw64-make -j$(nproc)
# - mingw64-make -j$(nproc) check
# tags:
# - shared
# - docker
# except:
# - tags
# artifacts:
# expire_in: 2 weeks
# when: on_failure
# paths:
# - ./*.log
# - tests/*.log

2
.gitmodules vendored
View File

@@ -1,3 +1,3 @@
[submodule "gnulib"]
path = gnulib
url = git://git.sv.gnu.org/gnulib.git
url = https://git.savannah.gnu.org/git/gnulib.git

7
.lgtm.yml Normal file
View File

@@ -0,0 +1,7 @@
extraction:
cpp:
prepare:
packages:
- libgpgme11-dev
configure:
command: ./bootstrap && ./configure

View File

@@ -1,4 +1,5 @@
sudo: no
dist: trusty
language: c

View File

@@ -1,7 +1,7 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
@@ -645,7 +645,7 @@ the "copyright" line and a pointer to where the full notice is found.
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
@@ -664,11 +664,11 @@ might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.
<https://www.gnu.org/philosophy/why-not-lgpl.html>.

View File

@@ -7,21 +7,14 @@ Primary List
The primary mailinglist for discussion, bug-reports, or questions about
GNU Wget is at <bug-wget@gnu.org>. To subscribe, send an email to
<bug-wget-join@gnu.org>, or visit
`http://lists.gnu.org/mailman/listinfo/bug-wget'.
`https://lists.gnu.org/mailman/listinfo/bug-wget'.
You do not need to subscribe to send a message to the list; however,
please note that unsubscribed messages are moderated, and may take a
while before they hit the list--*usually around a day*. If you want
your message to show up immediately, please subscribe to the list
before posting. Archives for the list may be found at
`http://lists.gnu.org/pipermail/bug-wget/'.
An NNTP/Usenettish gateway is also available via Gmane
(http://gmane.org/about.php). You can see the Gmane archives at
`http://news.gmane.org/gmane.comp.web.wget.general'. Note that the
Gmane archives conveniently include messages from both the current
list, and the previous one. Messages also show up in the Gmane archives
sooner than they do at `lists.gnu.org'.
`https://lists.gnu.org/pipermail/bug-wget/'.
Obsolete Lists
--------------
@@ -31,10 +24,7 @@ discussion list, and another list, <wget-patches@sunsite.dk> was used
for submitting and discussing patches to GNU Wget.
Messages from <wget@sunsite.dk> are archived at
`http://www.mail-archive.com/wget%40sunsite.dk/' and at
`http://news.gmane.org/gmane.comp.web.wget.general' (which also
continues to archive the current list, <bug-wget@gnu.org>).
`https://www.mail-archive.com/wget%40sunsite.dk/' and at
Messages from <wget-patches@sunsite.dk> are archived at
`http://news.gmane.org/gmane.comp.web.wget.patches'.
`https://news.gmane.org/gmane.comp.web.wget.patches'.

View File

@@ -1,6 +1,6 @@
# Makefile for `Wget' utility
# Copyright (C) 1995, 1996, 1997, 2006, 2007, 2008, 2009, 2010, 2011
# Free Software Foundation, Inc.
# Copyright (C) 1995-1997, 2006-2018 Free Software Foundation,
# Inc.
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -13,7 +13,7 @@
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# along with this program. If not, see <https://www.gnu.org/licenses/>.
# Additional permission under GNU GPL version 3 section 7
@@ -41,7 +41,7 @@ distuninstallcheck_listfiles = find . -type f | \
ACLOCAL_AMFLAGS = -I m4
# subdirectories in the distribution
SUBDIRS = lib src doc po tests util testenv
SUBDIRS = lib src doc po util fuzz tests testenv
EXTRA_DIST = MAILING-LIST \
msdos/config.h msdos/Makefile.DJ \
@@ -49,6 +49,7 @@ EXTRA_DIST = MAILING-LIST \
build-aux/build_info.pl build-aux/git-version-gen .version
CLEANFILES = *~ *.bak $(DISTNAME).tar.gz
DISTCLEANFILES = po/stamp-po
BUILT_SOURCES = .version

97
NEWS
View File

@@ -1,10 +1,103 @@
GNU Wget NEWS -- history of user-visible changes.
Copyright (C) 1997-2017 Free Software Foundation, Inc.
Copyright (C) 1997-2018 Free Software Foundation, Inc.
See the end for copying conditions.
Please send GNU Wget bug reports to <bug-wget@gnu.org>.
* Changes in Wget 1.20
** Add new option `--retry-on-host-error` to treat local errors as transient
and hence Wget will retry to download the file after a brief waiting period.
** Fixed multiple potential resource leaks as found by static analysis
** Wget will now not create an empty wget-log file when running with -q and -b
switches together
** When compiled using the GnuTLS >= 3.6.3, Wget now has support for TLSv1.3
** Now there is support for using libpcre2 for regex pattern matching
** When downloading over FTP recursively, one can now use the
--{accept,reject}-regex switches to fine-tune the downloaded files
** Building Wget from the git sources now requires autoconf 2.63 or above.
Building from the Tarballs works as it used to.
* Changes in Wget 1.19.5
* Fix cookie injection (CVE-2018-0494)
* Enable TLS1.3 with recent OpenSSL environment
* New option --ciphers to set GnuTLS / OpenSSL ciphers directly
* Updated CSS grammar to CSS 2.2
* Fixed several memleaks found by OSS-Fuzz
* Fixed several buffer overflows found by OSS-Fuzz
* Fixed several integer overflows found by OSS-Fuzz
* Several minor bug fixes
* Changes in Wget 1.19.4
* A major bug that caused GZip'ed pages to never be decompressed has been fixed
* Support for Content-Encoding and Transfer-Encoding have been marked as
experimental and disabled by default
* Changes in Wget 1.19.3
* Prevent erroneous decompression of .gz and .tgz files with broken servers
* Added support for HTTP 308 Permanent Redirect response
* Fix a segfault in some cases where the Content-Type header is not sent
* Support OpenSSL 1.1 builds without using deprecated features
* Fix netrc file detection on Windows
* Several minor bug fixes
* Changes in Wget 1.19.2
* Fix CVE-2017-13089 (Stack overflow in HTTP protocol handling)
* Fix CVE-2017-13090 (Heap overflow in HTTP protocol handling)
* New option --compression for gzip Content-Encoding
* New option --[no]-netrc to control .netrc parsing
* Added GNU extensions to .netrc parsing
* Improved IDNA 2003 compatibility
* Fix VPATH issues
* Improved and extended the test suite
* Support Wayback Machine's X-Archive-Orig-last-modified
* Several bug fixes
* Changes in Wget 1.19.1
* Fix bugs, a regression, portability/build issues
* Add new option --retry-on-http-error
* Changes in Wget 1.19
* New option --use-askpass=COMMAND. Fetch user/password by calling
@@ -691,7 +784,7 @@ addresses when accessing the first one fails.
non-standard port.
** Wget now supports the robots.txt directives specified in
<http://www.robotstxt.org/wc/norobots-rfc.txt>.
<http://www.robotstxt.org/norobots-rfc.txt>.
** URL parser has been fixed, especially the infamous overzealous
quoting. Wget no longer dequotes reserved characters, e.g. `%3F' is

6
README
View File

@@ -1,7 +1,7 @@
-*- text -*-
GNU Wget
========
Current Web home: http://www.gnu.org/software/wget/
Current Web home: https://www.gnu.org/software/wget/
GNU Wget is a free utility for non-interactive download of files from
the Web. It supports HTTP, HTTPS, and FTP protocols, as well as
@@ -50,10 +50,10 @@ resides at <ftp://ftp.gnu.org/pub/gnu/wget/>.
Please report bugs in Wget to <bug-wget@gnu.org>.
See the file `MAILING-LIST' for information about Wget mailing lists.
Wget's home page is at <http://www.gnu.org/software/wget/>.
Wget's home page is at <https://www.gnu.org/software/wget/>.
If you would like to contribute code for Wget, please read
http://wget.addictivecode.org/PatchGuidelines.
https://wget.addictivecode.org/PatchGuidelines.
Wget was originally written and mainained by Hrvoje Niksic. Please see
the file AUTHORS for a list of major contributors, and the ChangeLogs

View File

@@ -104,6 +104,10 @@ Compiling From Repository Sources
The command line options --dns-servers and --bind-dns-address are
only available when configured with --with-cares.
* [49]Gpef is required by some gnulib modules, most notably "unicase". You
need the application installed on the machine to compile Wget from git.
For those who might be confused as to what to do once they check out
the source code, considering configure and Makefile do not yet exist at
that point, a shell script called bootstrap has been provided. After
@@ -116,7 +120,7 @@ Compiling From Repository Sources
$ cd wget # assumes you've cloned a repository to "./wget"
2. Generate all the automatically-generated files required prior to
configuring the package:
$ ./bootstrap
$ ./bootstrap (or: bash ./bootstrap)
3. Configure the package and compile it:
$ ./configure --enable-assert [some_parameters]
$ make
@@ -157,7 +161,8 @@ Testing and development
that bug, which might be tedious. Tests should only be run through GDB for that purpose.
Copyright (C) 2008,2010,2014,2015 Free Software Foundation, Inc.
Copyright (C) 2008, 2010, 2014-2015, 2018 Free Software Foundation,
Inc.
License GPLv3+: GNU GPL version 3 or later
<https://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
@@ -198,7 +203,7 @@ References
32. https://www.gnu.org/software/libiconv/
33. https://git-scm.com/
34. https://github.com/rockdaboot/libpsl
35. http://www.pcre.org
35. https://www.pcre.org
36. http://wget.addictivecode.org/CompilingRepoSources?action=edit&editor=text
37. http://wget.addictivecode.org/CompilingRepoSources
38. http://wget.addictivecode.org/CompilingRepoSources?action=info
@@ -212,3 +217,4 @@ References
46. https://launchpad.net/libmetalink
47. https://www.gnupg.org
48. https://c-ares.haxx.se/
49. https://www.gnu.org/software/gperf/

331
bootstrap
View File

@@ -1,10 +1,10 @@
#! /bin/sh
# Print a version string.
scriptversion=2016-01-24.06; # UTC
scriptversion=2018-10-13.05; # UTC
# Bootstrap this package from checked-out sources.
# Copyright (C) 2003-2016 Free Software Foundation, Inc.
# Copyright (C) 2003-2018 Free Software Foundation, Inc.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -17,7 +17,7 @@ scriptversion=2016-01-24.06; # UTC
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# along with this program. If not, see <https://www.gnu.org/licenses/>.
# Originally written by Paul Eggert. The canonical version of this
# script is maintained as build-aux/bootstrap in gnulib, however, to
@@ -47,6 +47,8 @@ PERL="${PERL-perl}"
me=$0
default_gnulib_url=git://git.sv.gnu.org/gnulib
usage() {
cat <<EOF
Usage: $me [OPTION]...
@@ -76,6 +78,37 @@ contents are read as shell variables to configure the bootstrap.
For build prerequisites, environment variables like \$AUTOCONF and \$AMTAR
are honored.
Gnulib sources can be fetched in various ways:
* If this package is in a git repository with a 'gnulib' submodule
configured, then that submodule is initialized and updated and sources
are fetched from there. If \$GNULIB_SRCDIR is set (directly or via
--gnulib-srcdir) and is a git repository, then it is used as a reference.
* Otherwise, if \$GNULIB_SRCDIR is set (directly or via --gnulib-srcdir),
then sources are fetched from that local directory. If it is a git
repository and \$GNULIB_REVISION is set, then that revision is checked
out.
* Otherwise, if this package is in a git repository with a 'gnulib'
submodule configured, then that submodule is initialized and updated and
sources are fetched from there.
* Otherwise, if the 'gnulib' directory does not exist, Gnulib sources are
cloned into that directory using git from \$GNULIB_URL, defaulting to
$default_gnulib_url.
If \$GNULIB_REVISION is set, then that revision is checked out.
* Otherwise, the existing Gnulib sources in the 'gnulib' directory are
used. If it is a git repository and \$GNULIB_REVISION is set, then that
revision is checked out.
If you maintain a package and want to pin a particular revision of the
Gnulib sources that has been tested with your package, then there are two
possible approaches: either configure a 'gnulib' submodule with the
appropriate revision, or set \$GNULIB_REVISION (and if necessary
\$GNULIB_URL) in $me.conf.
Running without arguments will suffice in most cases.
EOF
}
@@ -129,19 +162,12 @@ bootstrap_post_import_hook() { :; }
# Override it via your own definition in bootstrap.conf.
bootstrap_epilogue() { :; }
# The command to download all .po files for a specified domain into
# a specified directory. Fill in the first %s is the domain name, and
# the second with the destination directory. Use rsync's -L and -r
# options because the latest/%s directory and the .po files within are
# all symlinks.
# The command to download all .po files for a specified domain into a
# specified directory. Fill in the first %s with the destination
# directory and the second with the domain name.
po_download_command_format=\
"rsync --delete --exclude '*.s1' -Lrtvz \
'translationproject.org::tp/latest/%s/' '%s'"
# Fallback for downloading .po files (if rsync fails).
po_download_command_format2=\
"wget --mirror -nd -q -np -A.po -P '%s' \
http://translationproject.org/latest/%s/"
"wget --mirror --level=1 -nd -q -A.po -P '%s' \
https://translationproject.org/latest/%s/"
# Prefer a non-empty tarname (4th argument of AC_INIT if given), else
# fall back to the package name (1st argument with munging)
@@ -170,7 +196,15 @@ source_base=lib
m4_base=m4
doc_base=doc
tests_base=tests
gnulib_extra_files=''
gnulib_extra_files="
build-aux/install-sh
build-aux/mdate-sh
build-aux/texinfo.tex
build-aux/depcomp
build-aux/config.guess
build-aux/config.sub
doc/INSTALL
"
# Additional gnulib-tool options to use. Use "\newline" to break lines.
gnulib_tool_option_extras=
@@ -264,24 +298,18 @@ case "$0" in
*) test -r "$0.conf" && . ./"$0.conf" ;;
esac
# Extra files from gnulib, which override files from other sources.
test -z "${gnulib_extra_files}" && \
gnulib_extra_files="
build-aux/install-sh
build-aux/mdate-sh
build-aux/texinfo.tex
build-aux/depcomp
build-aux/config.guess
build-aux/config.sub
doc/INSTALL
"
if test "$vc_ignore" = auto; then
vc_ignore=
test -d .git && vc_ignore=.gitignore
test -d CVS && vc_ignore="$vc_ignore .cvsignore"
fi
if test x"$gnulib_modules$gnulib_files$gnulib_extra_files" = x; then
use_gnulib=false
else
use_gnulib=true
fi
# Translate configuration into internal form.
# Parse options.
@@ -418,28 +446,30 @@ sort_ver() { # sort -V is not generally available
done
}
get_version_sed='
# Move version to start of line.
s/.*[v ]\([0-9]\)/\1/
# Skip lines that do not start with version.
/^[0-9]/!d
# Remove characters after the version.
s/[^.a-z0-9-].*//
# The first component must be digits only.
s/^\([0-9]*\)[a-z-].*/\1/
#the following essentially does s/5.005/5.5/
s/\.0*\([1-9]\)/.\1/g
p
q'
get_version() {
app=$1
$app --version >/dev/null 2>&1 || { $app --version; return 1; }
$app --version 2>&1 |
sed -n '# Move version to start of line.
s/.*[v ]\([0-9]\)/\1/
# Skip lines that do not start with version.
/^[0-9]/!d
# Remove characters after the version.
s/[^.a-z0-9-].*//
# The first component must be digits only.
s/^\([0-9]*\)[a-z-].*/\1/
#the following essentially does s/5.005/5.5/
s/\.0*\([1-9]\)/.\1/g
p
q'
$app --version 2>&1 | sed -n "$get_version_sed"
}
check_versions() {
@@ -610,95 +640,101 @@ git_modules_config () {
test -f .gitmodules && git config --file .gitmodules "$@"
}
if $use_git; then
gnulib_path=$(git_modules_config submodule.gnulib.path)
test -z "$gnulib_path" && gnulib_path=gnulib
fi
# Get gnulib files. Populate $GNULIB_SRCDIR, possibly updating a
# submodule, for use in the rest of the script.
case ${GNULIB_SRCDIR--} in
-)
# Note that $use_git is necessarily true in this case.
if git_modules_config submodule.gnulib.url >/dev/null; then
echo "$0: getting gnulib files..."
git submodule init -- "$gnulib_path" || exit $?
git submodule update -- "$gnulib_path" || exit $?
elif [ ! -d "$gnulib_path" ]; then
echo "$0: getting gnulib files..."
trap cleanup_gnulib 1 2 13 15
shallow=
git clone -h 2>&1 | grep -- --depth > /dev/null && shallow='--depth 2'
git clone $shallow git://git.sv.gnu.org/gnulib "$gnulib_path" ||
cleanup_gnulib
trap - 1 2 13 15
if $use_gnulib; then
if $use_git; then
gnulib_path=$(git_modules_config submodule.gnulib.path)
test -z "$gnulib_path" && gnulib_path=gnulib
fi
GNULIB_SRCDIR=$gnulib_path
;;
*)
# Use GNULIB_SRCDIR directly or as a reference.
if $use_git && test -d "$GNULIB_SRCDIR"/.git && \
git_modules_config submodule.gnulib.url >/dev/null; then
echo "$0: getting gnulib files..."
if git submodule -h|grep -- --reference > /dev/null; then
# Prefer the one-liner available in git 1.6.4 or newer.
git submodule update --init --reference "$GNULIB_SRCDIR" \
"$gnulib_path" || exit $?
else
# This fallback allows at least git 1.5.5.
if test -f "$gnulib_path"/gnulib-tool; then
# Since file already exists, assume submodule init already complete.
git submodule update -- "$gnulib_path" || exit $?
else
# Older git can't clone into an empty directory.
rmdir "$gnulib_path" 2>/dev/null
git clone --reference "$GNULIB_SRCDIR" \
"$(git_modules_config submodule.gnulib.url)" "$gnulib_path" \
&& git submodule init -- "$gnulib_path" \
&& git submodule update -- "$gnulib_path" \
|| exit $?
# Get gnulib files. Populate $GNULIB_SRCDIR, possibly updating a
# submodule, for use in the rest of the script.
case ${GNULIB_SRCDIR--} in
-)
# Note that $use_git is necessarily true in this case.
if git_modules_config submodule.gnulib.url >/dev/null; then
echo "$0: getting gnulib files..."
git submodule init -- "$gnulib_path" || exit $?
git submodule update -- "$gnulib_path" || exit $?
elif [ ! -d "$gnulib_path" ]; then
echo "$0: getting gnulib files..."
trap cleanup_gnulib 1 2 13 15
shallow=
if test -z "$GNULIB_REVISION"; then
git clone -h 2>&1 | grep -- --depth > /dev/null && shallow='--depth 2'
fi
git clone $shallow ${GNULIB_URL:-$default_gnulib_url} "$gnulib_path" \
|| cleanup_gnulib
trap - 1 2 13 15
fi
GNULIB_SRCDIR=$gnulib_path
;;
*)
# Use GNULIB_SRCDIR directly or as a reference.
if $use_git && test -d "$GNULIB_SRCDIR"/.git && \
git_modules_config submodule.gnulib.url >/dev/null; then
echo "$0: getting gnulib files..."
if git submodule -h|grep -- --reference > /dev/null; then
# Prefer the one-liner available in git 1.6.4 or newer.
git submodule update --init --reference "$GNULIB_SRCDIR" \
"$gnulib_path" || exit $?
else
# This fallback allows at least git 1.5.5.
if test -f "$gnulib_path"/gnulib-tool; then
# Since file already exists, assume submodule init already complete.
git submodule update -- "$gnulib_path" || exit $?
else
# Older git can't clone into an empty directory.
rmdir "$gnulib_path" 2>/dev/null
git clone --reference "$GNULIB_SRCDIR" \
"$(git_modules_config submodule.gnulib.url)" "$gnulib_path" \
&& git submodule init -- "$gnulib_path" \
&& git submodule update -- "$gnulib_path" \
|| exit $?
fi
fi
GNULIB_SRCDIR=$gnulib_path
fi
;;
esac
if test -d "$GNULIB_SRCDIR"/.git && test -n "$GNULIB_REVISION" \
&& ! git_modules_config submodule.gnulib.url >/dev/null; then
(cd "$GNULIB_SRCDIR" && git checkout "$GNULIB_REVISION") || cleanup_gnulib
fi
;;
esac
# $GNULIB_SRCDIR now points to the version of gnulib to use, and
# we no longer need to use git or $gnulib_path below here.
# $GNULIB_SRCDIR now points to the version of gnulib to use, and
# we no longer need to use git or $gnulib_path below here.
if $bootstrap_sync; then
cmp -s "$0" "$GNULIB_SRCDIR/build-aux/bootstrap" || {
echo "$0: updating bootstrap and restarting..."
case $(sh -c 'echo "$1"' -- a) in
a) ignored=--;;
*) ignored=ignored;;
esac
exec sh -c \
'cp "$1" "$2" && shift && exec "${CONFIG_SHELL-/bin/sh}" "$@"' \
$ignored "$GNULIB_SRCDIR/build-aux/bootstrap" \
"$0" "$@" --no-bootstrap-sync
}
if $bootstrap_sync; then
cmp -s "$0" "$GNULIB_SRCDIR/build-aux/bootstrap" || {
echo "$0: updating bootstrap and restarting..."
case $(sh -c 'echo "$1"' -- a) in
a) ignored=--;;
*) ignored=ignored;;
esac
exec sh -c \
'cp "$1" "$2" && shift && exec "${CONFIG_SHELL-/bin/sh}" "$@"' \
$ignored "$GNULIB_SRCDIR/build-aux/bootstrap" \
"$0" "$@" --no-bootstrap-sync
}
fi
gnulib_tool=$GNULIB_SRCDIR/gnulib-tool
<$gnulib_tool || exit $?
fi
gnulib_tool=$GNULIB_SRCDIR/gnulib-tool
<$gnulib_tool || exit $?
# Get translations.
download_po_files() {
subdir=$1
domain=$2
echo "$me: getting translations into $subdir for $domain..."
cmd=$(printf "$po_download_command_format" "$domain" "$subdir")
eval "$cmd" && return
# Fallback to HTTP.
cmd=$(printf "$po_download_command_format2" "$subdir" "$domain")
cmd=$(printf "$po_download_command_format" "$subdir" "$domain")
eval "$cmd"
}
@@ -788,9 +824,9 @@ symlink_to_dir()
# Leave any existing symlink alone, if it already points to the source,
# so that broken build tools that care about symlink times
# aren't confused into doing unnecessary builds. Conversely, if the
# existing symlink's time stamp is older than the source, make it afresh,
# existing symlink's timestamp is older than the source, make it afresh,
# so that broken tools aren't confused into skipping needed builds. See
# <http://lists.gnu.org/archive/html/bug-gnulib/2011-05/msg00326.html>.
# <https://lists.gnu.org/r/bug-gnulib/2011-05/msg00326.html>.
test -h "$dst" &&
src_ls=$(ls -diL "$src" 2>/dev/null) && set $src_ls && src_i=$1 &&
dst_ls=$(ls -diL "$dst" 2>/dev/null) && set $dst_ls && dst_i=$1 &&
@@ -896,32 +932,33 @@ fi
# Import from gnulib.
gnulib_tool_options="\
--import\
--no-changelog\
--aux-dir $build_aux\
--doc-base $doc_base\
--lib $gnulib_name\
--m4-base $m4_base/\
--source-base $source_base/\
--tests-base $tests_base\
--local-dir $local_gl_dir\
$gnulib_tool_option_extras\
"
if test $use_libtool = 1; then
case "$gnulib_tool_options " in
*' --libtool '*) ;;
*) gnulib_tool_options="$gnulib_tool_options --libtool" ;;
esac
fi
echo "$0: $gnulib_tool $gnulib_tool_options --import ..."
$gnulib_tool $gnulib_tool_options --import $gnulib_modules \
|| die "gnulib-tool failed"
if $use_gnulib; then
gnulib_tool_options="\
--no-changelog\
--aux-dir=$build_aux\
--doc-base=$doc_base\
--lib=$gnulib_name\
--m4-base=$m4_base/\
--source-base=$source_base/\
--tests-base=$tests_base\
--local-dir=$local_gl_dir\
$gnulib_tool_option_extras\
"
if test $use_libtool = 1; then
case "$gnulib_tool_options " in
*' --libtool '*) ;;
*) gnulib_tool_options="$gnulib_tool_options --libtool" ;;
esac
fi
echo "$0: $gnulib_tool $gnulib_tool_options --import ..."
$gnulib_tool $gnulib_tool_options --import $gnulib_modules \
|| die "gnulib-tool failed"
for file in $gnulib_files; do
symlink_to_dir "$GNULIB_SRCDIR" $file \
|| die "failed to symlink $file"
done
for file in $gnulib_files; do
symlink_to_dir "$GNULIB_SRCDIR" $file \
|| die "failed to symlink $file"
done
fi
bootstrap_post_import_hook \
|| die "bootstrap_post_import_hook failed"
@@ -1018,7 +1055,7 @@ bootstrap_epilogue
echo "$0: done. Now you can run './configure'."
# Local variables:
# eval: (add-hook 'write-file-hooks 'time-stamp)
# eval: (add-hook 'before-save-hook 'time-stamp)
# time-stamp-start: "scriptversion="
# time-stamp-format: "%:y-%02m-%02d.%02H"
# time-stamp-time-zone: "UTC0"

View File

@@ -1,6 +1,5 @@
# bootstrap.conf - Bootstrap configuration.
# Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012 Free Software Foundation,
# Inc.
# Copyright (C) 2007-2012, 2018 Free Software Foundation, Inc.
#
# This file is part of GNU Wget.
#
@@ -9,13 +8,13 @@
# the Free Software Foundation, either version 3 of the License, or (at
# your option) any later version.
#
# GNU Wgt is distributed in the hope that it will be useful, but
# GNU Wget is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see `http://www.gnu.org/licenses/'.
# along with this program. If not, see `https://www.gnu.org/licenses/'.
# gnulib library name
gnulib_name=libgnu
@@ -49,10 +48,11 @@ getopt-gnu
getpass-gnu
getpeername
getsockname
gettext
gettext-h
gitlog-to-changelog
git-version-gen
gnupload
group-member
hostent
ioctl
iconv
@@ -102,6 +102,7 @@ strerror_r-posix
strpbrk
strptime
strtok_r
strtol
strtoll
symlink
sys_types
@@ -111,11 +112,14 @@ unlink
unlocked-io
update-copyright
libunistring-optional
unistr/base
unicase/tolower
unistr/u8-strlen
unicase/u8-tolower
utime
utime-h
utimens
vasprintf
vsnprintf
wcwidth
write
xmemdup0
xstrndup
@@ -145,4 +149,21 @@ rsync -
tar -
xz -
flex -
gperf -
"
bootstrap_post_import_hook ()
{
# We need to patch unicase/special-casing-table.h for gperf < 3.1
GPERF_VER=$(gperf --version | {
IFS=' .' read x y major minor bugfix _
printf "%d%02d%02d" $major $minor $bugfix
})
if [ $GPERF_VER -lt 30100 ]; then
echo "Creating lib/unicase/special-casing-table.h for gperf < 3.1"
gperf -m 10 lib/unicase/special-casing-table.gperf | \
sed -e "s/gl_unicase_special_lookup.*/gl_unicase_special_lookup\ \(const char \*str, size_t len\)/g" \
> lib/unicase/special-casing-table.h
fi
}

View File

@@ -2,7 +2,7 @@
# Generate build_info.c.
# Copyright (C) 2009, 2010, 2011 Free Software Foundation, Inc.
# Copyright (C) 2009-2011, 2018 Free Software Foundation, Inc.
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by

12
cfg.mk
View File

@@ -1,4 +1,11 @@
export VC_LIST_EXCEPT_DEFAULT=^(lib/.*|m4/.*|md5/.*|build-aux/.*|src/gettext\.h|.*ChangeLog)$$
# Explicit syntax-check exceptions.
exclude_file_name_regexp--sc_trailing_blank = ^doc/annou.*$$
exclude_file_name_regexp--sc_bindtextdomain = ^fuzz/.*.c$$
exclude_file_name_regexp--sc_require_config_h = fuzz/main.c$$
exclude_file_name_regexp--sc_require_config_h_first = fuzz/main.c$$
export VC_LIST_EXCEPT_DEFAULT=^(lib/.*|m4/.*|md5/.*|build-aux/.*|src/gettext\.h|.*ChangeLog|tests/certs/.*)$$
local-checks-to-skip = \
sc_const_long_option \
@@ -21,9 +28,10 @@ local-checks-to-skip = \
sc_program_name \
sc_makefile_at_at_check \
sc_cast_of_argument_to_free \
sc_prohibit_undesirable_word_seq
#SHELL=bash -x
show-vc-list-except:
@$(VC_LIST_EXCEPT)
VC_LIST_ALWAYS_EXCLUDE_REGEX = ^ABOUT-NLS|((msdos|vms)/.*)|ChangeLog-2014-12-10|/.*.der$$
VC_LIST_ALWAYS_EXCLUDE_REGEX = ^ABOUT-NLS|((msdos|vms)/.*)|ChangeLog-2014-12-10|/.*\.der|^tests/certs/.*|^fuzz/.*\.(in|repro)/.*$$

View File

@@ -1,6 +1,6 @@
dnl Template file for GNU Autoconf
dnl Copyright (C) 1995, 1996, 1997, 2001, 2007, 2008, 2009, 2010, 2011, 2012,
dnl 2013, 2014 Free Software Foundation, Inc.
dnl Copyright (C) 1995-1997, 2001, 2007-2014, 2018 Free Software
dnl Foundation, Inc.
dnl This program is free software; you can redistribute it and/or modify
dnl it under the terms of the GNU General Public License as published by
@@ -33,7 +33,7 @@ dnl
AC_INIT([wget],
m4_esyscmd([build-aux/git-version-gen .tarball-version]),
[bug-wget@gnu.org])
AC_PREREQ(2.61)
AC_PREREQ(2.63)
dnl
dnl What version of Wget are we building?
@@ -87,6 +87,22 @@ dnl
dnl Process features
dnl
AC_ARG_ENABLE([fuzzing],
[AS_HELP_STRING([--enable-fuzzing], [Turn on fuzzing build (for developers)])],
[enable_fuzzing=yes;
AC_SUBST([LIB_FUZZING_ENGINE])
AC_DEFINE([FUZZING], 1, [Define to 1 if this is a fuzzing build])
], [enable_fuzzing=no; LIB_FUZZING_ENGINE=""])
#FUZZ_LIBS=$LIBS
#if test $enable_fuzzing = "yes"; then
# OLD_LIBS=$LIBS
# AC_SEARCH_LIBS([dlsym], [dl dld])
# FUZZ_LIBS=$LIBS
# LIBS=$OLD_LIBS
#fi
#AC_SUBST([FUZZ_LIBS])
AM_CONDITIONAL([FUZZING], [test "$enable_fuzzing" = "yes"])
dnl Opie: Support for opie s/key FTP logins
AC_ARG_ENABLE([opie],
[AS_HELP_STRING([--disable-opie], [disable support for opie or s/key FTP login])],
@@ -184,7 +200,7 @@ dnl
dnl Gettext
dnl
AM_GNU_GETTEXT([external],[need-ngettext])
AM_GNU_GETTEXT_VERSION([0.18.1])
AM_GNU_GETTEXT_VERSION([0.17])
AC_PROG_RANLIB
@@ -237,7 +253,7 @@ dnl
AC_HEADER_STDBOOL
AC_CHECK_HEADERS(unistd.h sys/time.h)
AC_CHECK_HEADERS(termios.h sys/ioctl.h sys/select.h utime.h sys/utime.h)
AC_CHECK_HEADERS(stdint.h inttypes.h pwd.h wchar.h)
AC_CHECK_HEADERS(stdint.h inttypes.h pwd.h wchar.h dlfcn.h)
AC_CHECK_DECLS(h_errno,,,[#include <netdb.h>])
@@ -275,7 +291,7 @@ AC_FUNC_MMAP
AC_FUNC_FSEEKO
AC_CHECK_FUNCS(strptime timegm vsnprintf vasprintf drand48 pathconf)
AC_CHECK_FUNCS(strtoll usleep ftello sigblock sigsetjmp memrchr wcwidth mbtowc)
AC_CHECK_FUNCS(sleep symlink utime strlcpy random)
AC_CHECK_FUNCS(sleep symlink utime strlcpy random fmemopen)
if test x"$ENABLE_OPIE" = xyes; then
AC_LIBOBJ([ftp-opie])
@@ -477,35 +493,6 @@ else
fi
fi
dnl
dnl Check for libmetalink
dnl
AS_IF([test x"$with_metalink" != xno], [
PKG_CHECK_MODULES([METALINK], libmetalink, [
LIBS="$METALINK_LIBS $LIBS"
CFLAGS="$METALINK_CFLAGS $CFLAGS"
AC_DEFINE([HAVE_METALINK], [1], [Define if using metalink.])
with_metalink=yes
], [
with_metalink=no
])
have_gpg=no
AS_IF([test x"$with_metalink" = xyes], [
dnl
dnl Check for GPGME
dnl
m4_ifdef([AM_PATH_GPGME], [
AM_PATH_GPGME([], [
LIBS="$GPGME_LIBS $LIBS"
CFLAGS="$GPGME_CFLAGS $CFLAGS"
AC_DEFINE([HAVE_GPGME], [1], [Define if GPGME is available.])
have_gpg=yes
])
])
])
])
dnl **********************************************************************
dnl Checks for IPv6
dnl **********************************************************************
@@ -641,10 +628,6 @@ AS_IF([test "X$iri" != "Xno"],[
AC_MSG_NOTICE([disabling IRIs because libiconv wasn't found])
fi
fi
],[ # else
# For some reason, this seems to be set even when we don't check.
# Explicitly unset.
LIBICONV=
])
AC_ARG_WITH(libidn, AC_HELP_STRING([--with-libidn=[DIR]],
@@ -663,11 +646,6 @@ if test "X$iri" != "Xno"; then
# ])
])
fi
if test "X$iri" = "Xno"; then
# we don't need libunistring - clear settings from gnulib module
LIBUNISTRING=""
LTLIBUNISTRING=""
fi
dnl
dnl Check for UUID
@@ -708,18 +686,39 @@ AS_IF([test "x$uuid_mode" = x1], [
])
dnl
dnl Check for PCRE
dnl Check for PCRE2 / PCRE
dnl
AC_ARG_ENABLE(pcre2, AC_HELP_STRING([--disable-pcre2],
[Disable PCRE2 style regular expressions]))
AC_ARG_ENABLE(pcre, AC_HELP_STRING([--disable-pcre],
[Disable PCRE style regular expressions]))
AS_IF([test "X$enable_pcre" != "Xno"],[
AS_IF([test "X$enable_pcre2" != "Xno"],[
enable_pcre2=no
PKG_CHECK_MODULES([PCRE2], libpcre2-8, [
CFLAGS="$PCRE2_CFLAGS $CFLAGS"
LIBS="$PCRE2_LIBS $LIBS"
AC_DEFINE([HAVE_LIBPCRE2], [1], [Define if libpcre2 is available.])
enable_pcre2=yes
], [
AC_CHECK_HEADER(pcre2.h, [
AC_CHECK_LIB(pcre2-8, pcre2_compile_8, [
LIBS="${LIBS} -lpcre2-8"
AC_DEFINE([HAVE_LIBPCRE2], 1, [Define if libpcre2 is available.])
enable_pcre2=yes
])
])
])
])
AS_IF([test "X$enable_pcre" != "Xno" && test "X$enable_pcre2" != "Xyes"],[
PKG_CHECK_MODULES([PCRE], libpcre, [
CFLAGS="$PCRE_CFLAGS $CFLAGS"
AC_CHECK_HEADER(pcre.h, [
LIBS="$PCRE_LIBS $LIBS"
AC_DEFINE([HAVE_LIBPCRE], [1], [Define if libpcre is available.])
enable_pcre=yes
])
], [
AC_CHECK_HEADER(pcre.h, [
@@ -731,6 +730,8 @@ AS_IF([test "X$enable_pcre" != "Xno"],[
])
])
AS_IF([test "X$enable_pcre2" = Xyes], [PCRE_INFO="yes, via libpcre2"], [test "X$enable_pcre" = Xyes], [PCRE_INFO="yes, via libpcre"], [PCRE_INFO=no])
dnl
dnl Check for libcares (resolver library)
dnl
@@ -756,6 +757,38 @@ AS_IF([test "X$with_cares" = "Xyes"],[
RESOLVER_INFO="libc, --bind-dns-address and --dns-servers not available"
])
dnl
dnl Check for libmetalink
dnl
AS_IF([test x"$with_metalink" != xno], [
PKG_CHECK_MODULES([METALINK], libmetalink, [
LIBS="$METALINK_LIBS $LIBS"
CFLAGS="$METALINK_CFLAGS $CFLAGS"
AC_DEFINE([HAVE_METALINK], [1], [Define if using metalink.])
with_metalink=yes
], [
with_metalink=no
])
have_gpg=no
AS_IF([test x"$with_metalink" = xyes], [
dnl
dnl Check for GPGME
dnl
m4_ifdef([AM_PATH_GPGME], [
AM_PATH_GPGME([], [
# Put libgpgme to the end of the library list since it introduces a -L linker flags.
# That -L might break the build if there are two different version of
# a library (e.g. GnuTLS) in /usr/local and in the system directory.
LIBS="$LIBS $GPGME_LIBS"
CFLAGS="$GPGME_CFLAGS $CFLAGS"
AC_DEFINE([HAVE_GPGME], [1], [Define if GPGME is available.])
have_gpg=yes
])
])
])
])
dnl
dnl Extended Attribute support
dnl
@@ -792,8 +825,9 @@ dnl
dnl Create output
dnl
AC_CONFIG_FILES([Makefile src/Makefile doc/Makefile util/Makefile
po/Makefile.in tests/Makefile
lib/Makefile testenv/Makefile])
po/Makefile.in tests/Makefile fuzz/Makefile
lib/Makefile testenv/Makefile
tests/certs/interca.conf tests/certs/rootca.conf])
AC_CONFIG_HEADERS([src/config.h])
AC_OUTPUT
@@ -809,6 +843,7 @@ AC_MSG_NOTICE([Summary of build options:
SSL: $with_ssl
Zlib: $with_zlib
PSL: $with_libpsl
PCRE: $PCRE_INFO
Digest: $ENABLE_DIGEST
NTLM: $ENABLE_NTLM
OPIE: $ENABLE_OPIE
@@ -820,4 +855,5 @@ AC_MSG_NOTICE([Summary of build options:
Resolver: $RESOLVER_INFO
GPGME: $have_gpg
IRI: $iri
Fuzzing build: $enable_fuzzing, $LIB_FUZZING_ENGINE
])

View File

@@ -12,7 +12,8 @@ if [ $CORES -lt 1 ]; then CORES=1; fi
echo "Running: make distclean"
make distclean > /dev/null || true
CFLAGS="-std=c89 -pedantic -O0 -g -Wall -Wextra -Wstrict-prototypes -Wold-style-definition -Wwrite-strings -Wshadow -Wformat -Wformat-security -Wunreachable-code -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition"
#CFLAGS="-std=c89 -pedantic -O0 -g -Wall -Wextra -Wstrict-prototypes -Wold-style-definition -Wwrite-strings -Wshadow -Wformat -Wformat-security -Wunreachable-code -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition"
CFLAGS="-Wall -Wextra -O2 -g"
CACHEFILE=$PWD/config_check.cache
@@ -30,7 +31,7 @@ for CC in gcc clang; do
# the compiler changed, so we have to remove the cache file here
rm -f $CACHEFILE
for options in "" "--with-ssl=openssl"; do
for options in "" "--with-ssl=openssl" "--disable-iri"; do
export DISTCHECK_CONFIGURE_FLAGS="-C --cache-file=$CACHEFILE $options"
echo " ./configure $DISTCHECK_CONFIGURE_FLAGS"
./configure $DISTCHECK_CONFIGURE_FLAGS CFLAGS="$CFLAGS" > /dev/null

113
contrib/make-release Executable file
View File

@@ -0,0 +1,113 @@
#!/usr/bin/env sh
# GNU Wget: Make a new release
#
# This is a helper script to make a new release of GNU Wget
#
# Make sure that
set -e
set -o pipefail
set -u
export CFLAGS="-g -O2 -Wall"
EXTENSIONS="gzip lzip"
ask_continue() {
printf "Continue? [y/N]: "
read inp
[ "$inp" != "y" ] && [ "$inp" != "Y" ] && exit 1
# For some completely weird reason, this set +x is required. Else, the test
# *ALWAYS* fails
set +x
}
if [ ! -d ".git" ]; then
echo "Please run this script from the root of the git repository"
fi
echo "This script will make a new release and upload it to the GNU FTP Servers"
echo "Will run git clean -dxf to get a clean dir"
ask_continue
# Make sure we have a clean working directory
git clean -dxfq
echo "=> Bootstrap"
./bootstrap > /dev/null 2>&1
echo "=> Configure"
./configure -q
APP_NAME=$(grep "^PACKAGE_NAME='" configure | cut -d "'" -f2)
APP_VER=$(grep "^PACKAGE_VERSION='" configure | cut -d "'" -f2)
echo "Making release for $APP_NAME $APP_VER"
ask_continue
NEWS_VERSION=$(grep -i "Changes in $APP_NAME" NEWS | head -1 | awk '{print $5}')
if [ "$NEWS_VERSION" != "$APP_VER" ]; then
echo "Latest version in NEWS file does not match latest tag"
exit 1
fi
CUR_VER=$(git describe | sed 's/^v//')
PREV_VER=$(git describe --abbrev=0 --tags $(git rev-list --tags --skip=1 --max-count=1) | sed 's/^v//')
LNO_CVER=$(grep -in "Changes in $APP_NAME" NEWS | head -1 | cut -f1 -d:)
LNO_PVER=$(grep -in "Changes in $APP_NAME" NEWS | head -2 | tail -1 | cut -f1 -d:)
sed -n "${LNO_CVER},${LNO_PVER}p" NEWS
echo "This is the current contents of your NEWS"
ask_continue
echo "=> Make distcheck"
make -s distcheck > /dev/null 2>&1
for ext in $EXTENSIONS; do
[ "$ext" = 'gz' ] && ext="gzip"
[ "$ext" = 'lz' ] && ext="lzip"
echo "=> Making $APP_NAME-$CUR_VER.tar.$ext"
make -s "dist-$ext" > /dev/null
done
echo ""
REL_TYPE=
REL_LOC=
while [ -z $REL_TYPE ]; do
printf "Enter release type (alpha, beta, stable): "
read x
case $x in
a|A|alpha|Alpha) REL_TYPE=alpha && REL_LOC="alpha.gnu.org";;
b|B|beta|Beta) REL_TYPE=beta && REL_LOC="alpha.gnu.org";;
s|S|stable|Stable) REL_TYPE=stable && REL_LOC="ftp.gnu.org";;
esac
done
GPG_KEY=
while [ -z $GPG_KEY ]; do
printf "Enter GPG Key ID for Signing: "
read key
gpg --list-keys "$key" || continue
printf "Will use this key to sign releases. Continue? [y/N]: "
read x
[ "$x" = "y" ] || [ "$x" = "Y" ] && GPG_KEY="$key"
done
echo ""
APP_LOWER="$(echo "$APP_NAME" | tr '[:upper:]' '[:lower:]')"
./build-aux/gnupload --to "$REL_LOC:$APP_LOWER" --user "$GPG_KEY" --symlink-regex "$APP_LOWER-$CUR_VER".tar.*
echo ""
build-aux/announce-gen --release-type "$REL_TYPE" --package-name "$APP_LOWER" --previous-version="$PREV_VER" --current-version "$CUR_VER" --gpg-key-id="$GPG_KEY" --url-directory "https://${REL_LOC}/gnu/${APP_LOWER}" --news=NEWS > "$APP_NAME-$APP_VER.announce"
echo ""
echo "Announce email template placed at $APP_NAME-$APP_VER.announce"
prev_tag=$(git tag | tail -2 | head -1)
echo ""
echo "Don't forget to mention the following contributors in the announcement:"
(
git log "${prev_tag}.." --format="%aN" --reverse
git log "${prev_tag}.." --reverse | grep -i reported | cut -d':' -f2 | cut -d'<' -f1 | cut -d' ' -f2- | tr -d '"'
) | sort -u

123
contrib/release Executable file
View File

@@ -0,0 +1,123 @@
#!/bin/bash -e
#
# Written 2016 by Tim Ruehsen
#
# To get the version number right: add a signed tag and
# make sure the working dir is clean (git diff).
# If that fails, put the version into '.tarball-version'.
export CFLAGS="-g -O2 -Wall"
echo
echo "* Bootstrap"
#./bootstrap >/dev/null
echo
echo "* Configure"
#./configure -q
extensions="gz lz"
app_name=$(grep "^PACKAGE_NAME='" configure|cut -d "'" -f2)
app_ver=$(grep "^PACKAGE_VERSION='" configure|cut -d "'" -f2)
echo "Current version of $app_name is $app_ver"
echo
echo -n "Is that OK ? [y/N] "
read x
[ "$x" != "y" -a "$x" != "Y" ] && exit 0
echo
grep -i "Changes in $app_name" NEWS|head -1
echo -n "Did you edit the NEWS file ? [y/N] "
read x
[ "$x" != "y" -a "$x" != "Y" ] && exit 0
echo
echo "* Make"
#make -s clean
#make -s
echo
echo "* Make distcheck"
#make -s distcheck >/dev/null
echo
echo "* Make dist"
for ext in $extensions; do
[[ $ext = 'gz' ]] && ext="gzip"
[[ $ext = 'lz' ]] && ext="lzip"
make -s dist-$ext >/dev/null
done
while true; do
echo "Alpha or Regular release ? [R/A] "
read x
if test "$x" = "r" || test "$x" = "R"; then
directory="/incoming/ftp"
break
elif test "$x" = "a" || test "$x" = "A"; then
directory="/incoming/alpha"
break
fi
done
x=$(gpg --list-secret-keys 2>/dev/null|grep ^sec|awk '{ print $2 }')
echo $x
if [ "$x" = "" ]; then
echo No signing key found - please check with 'gpg --list-secret-keys'
exit 1
fi
# arrays just work in Bash
keys=($x)
if [ ${#keys[@]} -eq 1 ]; then
IFS='/' arr=($x)
signing_key=${arr[1]}
else
for ((i=0; i < ${#keys[@]}; i++)); do
echo "$((i+1))) ${keys[i]}"
done
while true; do
echo -n "Enter 1-${#keys[@]}: "
read x
# check for digits only
if [ -n "${i//[0-9]/}" ]; then continue; fi
if test $x -ge 1 || test $x -le ${#keys[@]}; then
IFS='/' arr=(${keys[$((x-1))]})
signing_key=${arr[1]}
break
fi
done
fi
unset IFS
echo
echo -n "Really upload to ftp-upload.gnu.org/$directory ? "
read x
[ "$x" != "y" -a "$x" != "Y" ] && exit 0
for ext in $extensions; do
app_tarball="${app_name}-${app_ver}.tar.$ext"
gpg --default-key $signing_key --yes -b $app_tarball
echo -e "version: 1.2\nfilename: ${app_tarball}\ndirectory: ${app_name}\n" >${app_tarball}.directive
gpg --default-key $signing_key --yes --clearsign ${app_tarball}.directive
ftp -v -n -i -p ftp-upload.gnu.org <<EOF
user anonymous
cd $directory
put ${app_tarball}
put ${app_tarball}.sig
put ${app_tarball}.directive.asc
EOF
done
prev_tag=`git tag|tail -2|head -1`
echo "Don't forget to mention the following contributors in the announcement:"
(
git log ${prev_tag}.. --format="%aN" --reverse
git log ${prev_tag}.. --reverse|grep -i reported|cut -d':' -f2|cut -d'<' -f1|cut -d' ' -f2-|tr -d '"'
) | sort -u

View File

@@ -1,6 +1,6 @@
# Makefile for `wget' utility
# Copyright (C) 1995, 1996, 1997, 2007, 2008, 2009, 2010, 2011, 2015
# Free Software Foundation, Inc.
# Copyright (C) 1995-1997, 2007-2011, 2015, 2018 Free Software
# Foundation, Inc.
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by

104
doc/announcement_1.19.txt Normal file
View File

@@ -0,0 +1,104 @@
Hello,
we are pleased to announce the new version of GNU wget 1.19.
It comes with major improvements for Metalink, IDNA2008 for
international domain names, an option to call external tools
for fetching user/password, several bugfixes and improvements.
Many thanks go to all the contributors and list activists !
Contributors (from the git log):
Ander Juaristi
Dale R. Worley
Eli Zaretskii
Gisle Vanem
Giuseppe Scrivano
Göran Uddeborg
Jann Horn
Jeffery To
Jérémie Courrèges-Anglas
Liam R. Howlett
losgrandes
Matthew White
Misra, Deapesh
Nikos Mavrogiannopoulos
Noël Köthe
Piotr Wajda
Rahul Bedarkar
Sean Burford
Tim Rühsen
Tobias Stoeckmann
vijeth-aradhya
The new version is available for download here:
https://ftp.gnu.org/gnu/wget/wget-1.19.tar.gz
https://ftp.gnu.org/gnu/wget/wget-1.19.tar.xz
and the GPG detached signatures using the key 0x08302DB6A2670428:
https://ftp.gnu.org/gnu/wget/wget-1.19.tar.gz.sig
https://ftp.gnu.org/gnu/wget/wget-1.19.tar.xz.sig
To reduce load on the main server, you can use this redirector service
which automatically redirects you to a mirror:
http://ftpmirror.gnu.org/wget/wget-1.19.tar.gz
http://ftpmirror.gnu.org/wget/wget-1.19.tar.xz
Noteworthy changes:
* New option --use-askpass=COMMAND. Fetch user/password by calling
an external program.
* Use IDNA2008 (+ TR46 if available) through libidn2
* When processing a Metalink header, --metalink-index=<number> allows
to process the header's application/metalink4+xml files.
* When processing a Metalink file, --trust-server-names enables the
use of the destination file names specified in the Metalink file,
otherwise a safe destination file name is computed.
* When processing a Metalink file, enforce a safe destination path.
Remove any drive letter prefix under w32, i.e. 'C:D:file'. Call
libmetalink's metalink_check_safe_path() to prevent absolute,
relative, or home paths:
https://tools.ietf.org/html/rfc5854#section-4.1.2.1
https://tools.ietf.org/html/rfc5854#section-4.2.8.3
* When processing a Metalink file, --directory-prefix=<prefix> sets
the top of the retrieval tree to prefix for Metalink downloads.
* When processing a Metalink file, reject downloaded files which don't
agree with their own metalink:size value:
https://tools.ietf.org/html/rfc5854#section-4.2.16
* When processing a Metalink file, with --continue resume partially
downloaded files and keep fully downloaded files even if they fail
the verification.
* When processing a Metalink file, create the parent directories of a
"path/file" destination file name:
https://tools.ietf.org/html/rfc5854#section-4.1.2.1
https://tools.ietf.org/html/rfc5854#section-4.2.8.3
* On a recursive download, append a .tmp suffix to temporary files
that will be deleted after being parsed, and create them
readable/writable only by the owner.
* New make target 'check-valgrind'
* Fix several bugs
* Fix compatibility issues
Please report any problem you may experience to the bug-wget@gnu.org
mailing list.
For the maintainers of Wget,
Tim

View File

@@ -5,7 +5,7 @@
@c hence no sectioning command or @node.
@display
Copyright @copyright{} 2000, 2001, 2002, 2007, 2008, 2015 Free Software
Copyright @copyright{} 2000-2002, 2007-2008, 2015, 2018 Free Software
Foundation, Inc.
@uref{http://fsf.org/}

View File

@@ -1,7 +1,7 @@
#! /usr/bin/env perl
# Copyright (C) 1999, 2000, 2001, 2003, 2007, 2009, 2010, 2011, 2015
# Free Software Foundation, Inc.
# Copyright (C) 1999-2001, 2003, 2007, 2009-2011, 2015, 2018 Free
# Software Foundation, Inc.
# This file is part of GCC.

View File

@@ -31,9 +31,8 @@ This file documents the GNU Wget utility for downloading network
data.
@c man begin COPYRIGHT
Copyright @copyright{} 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
Foundation, Inc.
Copyright @copyright{} 1996-2011, 2015, 2018 Free Software Foundation,
Inc.
@iftex
Permission is granted to make and distribute verbatim copies of
@@ -193,6 +192,8 @@ options, or via the initialization file @file{.wgetrc} (@pxref{Startup
File}). Wget allows you to define @dfn{global} startup files
(@file{/usr/local/etc/wgetrc} by default) for site settings. You can also
specify the location of a startup file with the --config option.
To disable the reading of config files, use --no-config.
If both --config and --no-config are given, --no-config is ignored.
@ignore
@@ -566,7 +567,10 @@ would be resolved to @samp{http://foo/baz/b.html}.
@cindex specify config
@item --config=@var{FILE}
Specify the location of a startup file you wish to use.
Specify the location of a startup file you wish to use instead of the
default one(s). Use --no-config to disable reading of config files.
If both --config and --no-config are given, --no-config is ignored.
@item --rejected-log=@var{logfile}
Logs all URL rejections to @var{logfile} as comma separated values. The values
@@ -703,6 +707,12 @@ Before (over)writing a file, back up an existing file by adding a
files are rotated to @samp{.2}, @samp{.3}, and so on, up to
@var{backups} (and lost beyond that).
@cindex authentication credentials
@item --no-netrc
Do not try to obtain credentials from @file{.netrc} file. By default
@file{.netrc} file is searched for credentials in case none have been
passed on command line and authentication is required.
@cindex continue retrieval
@cindex incomplete downloads
@cindex resume download
@@ -1335,6 +1345,11 @@ renamed from @samp{--html-extension}, to better reflect its new
behavior. The old option name is still acceptable, but should now be
considered deprecated.
As of version 1.19.2, Wget will also ensure that any downloaded files with
a @code{Content-Encoding} of @samp{br}, @samp{compress}, @samp{deflate}
or @samp{gzip} end in the suffix @samp{.br}, @samp{.Z}, @samp{.zlib}
and @samp{.gz} respectively.
At some point in the future, this option may well be expanded to
include suffixes for other types of content, including content types
that are not parsed by Wget.
@@ -1519,6 +1534,23 @@ wget --header="Host: foo.bar" http://localhost/
In versions of Wget prior to 1.10 such use of @samp{--header} caused
sending of duplicate headers.
@cindex Content-Encoding, choose
@item --compression=@var{type}
Choose the type of compression to be used. Legal values are
@samp{auto}, @samp{gzip} and @samp{none}.
If @samp{auto} or @samp{gzip} are specified, Wget asks the server to
compress the file using the gzip compression format. If the server
compresses the file and responds with the @code{Content-Encoding}
header field set appropriately, the file will be decompressed
automatically.
If @samp{none} is specified, wget will not ask the server to compress
the file and will not decompress any server responses. This is the default.
Compression support is currently experimental. In case it is turned on,
please report any bugs to @code{bug-wget@@gnu.org}.
@cindex redirect
@item --max-redirect=@var{number}
Specifies the maximum number of redirections to follow for a resource.
@@ -1718,6 +1750,25 @@ some few obscure servers, which never send HTTP authentication
challenges, but accept unsolicited auth info, say, in addition to
form-based authentication.
@item --retry-on-host-error
Consider host errors, such as ``Temporary failure in name resolution'',
as non-fatal, transient errors.
@item --retry-on-http-error=@var{code[,code,...]}
Consider given HTTP response codes as non-fatal, transient errors.
Supply a comma-separated list of 3-digit HTTP response codes as
argument. Useful to work around special circumstances where retries
are required, but the server responds with an error code normally not
retried by Wget. Such errors might be 503 (Service Unavailable) and
429 (Too Many Requests). Retries enabled by this option are performed
subject to the normal retry timing and retry count limitations of
Wget.
Using this option is intended to support special use cases only and is
generally not recommended, as it can force retries even in cases where
the server is actually trying to decrease its load. Please use wisely
and only if you know what you are doing.
@end table
@node HTTPS (SSL/TLS) Options, FTP Options, HTTP Options, Invoking
@@ -1733,25 +1784,34 @@ If Wget is compiled without SSL support, none of these options are available.
@cindex SSL protocol, choose
@item --secure-protocol=@var{protocol}
Choose the secure protocol to be used. Legal values are @samp{auto},
@samp{SSLv2}, @samp{SSLv3}, @samp{TLSv1}, @samp{TLSv1_1}, @samp{TLSv1_2}
and @samp{PFS}. If @samp{auto} is used, the SSL library is given the
liberty of choosing the appropriate protocol automatically, which is
@samp{SSLv2}, @samp{SSLv3}, @samp{TLSv1}, @samp{TLSv1_1}, @samp{TLSv1_2},
@samp{TLSv1_3} and @samp{PFS}. If @samp{auto} is used, the SSL library is
given the liberty of choosing the appropriate protocol automatically, which is
achieved by sending a TLSv1 greeting. This is the default.
Specifying @samp{SSLv2}, @samp{SSLv3}, @samp{TLSv1}, @samp{TLSv1_1} or
@samp{TLSv1_2} forces the use of the corresponding protocol. This is
useful when talking to old and buggy SSL server implementations that
make it hard for the underlying SSL library to choose the correct
protocol version. Fortunately, such servers are quite rare.
Specifying @samp{SSLv2}, @samp{SSLv3}, @samp{TLSv1}, @samp{TLSv1_1},
@samp{TLSv1_2} or @samp{TLSv1_3} forces the use of the corresponding
protocol. This is useful when talking to old and buggy SSL server
implementations that make it hard for the underlying SSL library to choose
the correct protocol version. Fortunately, such servers are quite rare.
Specifying @samp{PFS} enforces the use of the so-called Perfect Forward
Security cipher suites. In short, PFS adds security by creating a one-time
key for each SSL connection. It has a bit more CPU impact on client and server.
We use known to be secure ciphers (e.g. no MD4) and the TLS protocol.
We use known to be secure ciphers (e.g. no MD4) and the TLS protocol. This mode
also explicitly excludes non-PFS key exchange methods, such as RSA.
@item --https-only
When in recursive mode, only HTTPS links are followed.
@item --ciphers
Set the cipher list string. Typically this string sets the
cipher suites and other SSL/TLS options that the user wish should be used, in a
set order of preference (GnuTLS calls it 'priority string'). This string
will be fed verbatim to the SSL/TLS engine (OpenSSL or GnuTLS) and hence
its format and syntax is dependant on that. Wget will not process or manipulate it
in any way. Refer to the OpenSSL or GnuTLS documentation for more information.
@cindex SSL certificate, check
@item --no-check-certificate
Don't check the server certificate against the available certificate
@@ -3337,10 +3397,16 @@ Turn globbing on/off---the same as @samp{--glob} and @samp{--no-glob}.
Define a header for HTTP downloads, like using
@samp{--header=@var{string}}.
@item compression = @var{string}
Choose the compression type to be used. Legal values are @samp{auto}
(the default), @samp{gzip}, and @samp{none}. The same as
@samp{--compression=@var{string}}.
@item adjust_extension = on/off
Add a @samp{.html} extension to @samp{text/html} or
@samp{application/xhtml+xml} files that lack one, or a @samp{.css}
extension to @samp{text/css} files that lack one, like
@samp{application/xhtml+xml} files that lack one, a @samp{.css}
extension to @samp{text/css} files that lack one, and a @samp{.br},
@samp{.Z}, @samp{.zlib} or @samp{.gz} to compressed files like
@samp{-E}. Previously named @samp{html_extension} (still acceptable,
but deprecated).
@@ -4080,7 +4146,11 @@ via IRC at @code{irc.freenode.org}, @code{#wget}. Come check it out!
@c man begin BUGS
You are welcome to submit bug reports via the GNU Wget bug tracker (see
@url{https://savannah.gnu.org/bugs/?func=additem&group=wget}).
@url{https://savannah.gnu.org/bugs/?func=additem&group=wget}) or to our
mailing list @email{bug-wget@@gnu.org}.
Visit @url{https://lists.gnu.org/mailman/listinfo/bug-wget} to
get more info (how to subscribe, list archives, ...).
Before actually submitting a bug report, please try to follow a few
simple guidelines.
@@ -4251,7 +4321,7 @@ server.
Until version 1.8, Wget supported the first version of the standard,
written by Martijn Koster in 1994 and available at
@url{http://www.robotstxt.org/robotstxt.html}. As of version 1.8,
@url{http://www.robotstxt.org/orig.html}. As of version 1.8,
Wget has supported the additional directives specified in the internet
draft @samp{<draft-koster-robots-00.txt>} titled ``A Method for Web
Robots Control''. The draft, which has as far as I know never made to
@@ -4270,7 +4340,7 @@ this:
@end example
This is explained in some detail at
@url{http://www.robotstxt.org/wc/meta-user.html}. Wget supports this
@url{http://www.robotstxt.org/meta.html}. Wget supports this
method of robot exclusion in addition to the usual @file{/robots.txt}
exclusion.

87
fuzz/Makefile.am Normal file
View File

@@ -0,0 +1,87 @@
AM_CFLAGS = $(WERROR_CFLAGS) $(WARN_CFLAGS) -Wno-unused-parameter -Wno-pedantic
AM_CPPFLAGS = -I$(top_srcdir)/src -I$(srcdir) -I$(top_builddir)/lib -I$(top_srcdir)/lib \
-DSRCDIR=\"$(abs_srcdir)\"
LDADD = ../lib/libgnu.a \
$(GETADDRINFO_LIB) $(HOSTENT_LIB) $(INET_NTOP_LIB) $(INET_PTON_LIB) \
$(LIBSOCKET) $(LIB_CLOCK_GETTIME) $(LIB_CRYPTO) $(LIB_GETLOGIN) $(LIB_NANOSLEEP) $(LIB_POLL) \
$(LIB_POSIX_SPAWN) $(LIB_PTHREAD_SIGMASK) $(LIB_SELECT) $(LIBICONV) $(LIBINTL) \
$(LIBMULTITHREAD) $(LIBTHREAD) $(SERVENT_LIB) @INTL_MACOSX_LIBS@ \
-ldl
WGET_TESTS = \
wget_css_fuzzer$(EXEEXT) \
wget_cookie_fuzzer$(EXEEXT) \
wget_ftpls_fuzzer$(EXEEXT) \
wget_html_fuzzer$(EXEEXT) \
wget_netrc_fuzzer$(EXEEXT) \
wget_options_fuzzer$(EXEEXT) \
wget_robots_fuzzer$(EXEEXT) \
wget_url_fuzzer$(EXEEXT)
if FUZZING
bin_PROGRAMS = $(WGET_TESTS)
LDADD += $(LIB_FUZZING_ENGINE)
MAIN = fuzzer.h
# AM_LDFLAGS = -no-install -all-static
else
AM_CPPFLAGS += -DTEST_RUN
AM_TESTS_ENVIRONMENT = export VALGRIND_TESTS"=@VALGRIND_TESTS@";
TESTS = $(WGET_TESTS)
check_PROGRAMS = $(WGET_TESTS)
MAIN = main.c fuzzer.h
endif
# Make libunittest "PHONY" so we're always sure we're up-to-date.
.PHONY: ../src/libunittest.a
../src/libunittest.a:
$(MAKE) $(AM_MAKEFLAGS) -C ../src libunittest.a
wget_css_fuzzer_SOURCES = wget_css_fuzzer.c $(MAIN)
wget_css_fuzzer_LDADD = ../src/libunittest.a $(LDADD)
wget_cookie_fuzzer_SOURCES = wget_cookie_fuzzer.c $(MAIN)
wget_cookie_fuzzer_LDADD = ../src/libunittest.a $(LDADD)
wget_ftpls_fuzzer_SOURCES = wget_ftpls_fuzzer.c $(MAIN)
wget_ftpls_fuzzer_LDADD = ../src/libunittest.a $(LDADD)
wget_html_fuzzer_SOURCES = wget_html_fuzzer.c $(MAIN)
wget_html_fuzzer_LDADD = ../src/libunittest.a $(LDADD)
wget_netrc_fuzzer_SOURCES = wget_netrc_fuzzer.c $(MAIN)
wget_netrc_fuzzer_LDADD = ../src/libunittest.a $(LDADD)
wget_options_fuzzer_SOURCES = wget_options_fuzzer.c $(MAIN)
wget_options_fuzzer_LDADD = ../src/libunittest.a $(LDADD)
wget_robots_fuzzer_SOURCES = wget_robots_fuzzer.c $(MAIN)
wget_robots_fuzzer_LDADD = ../src/libunittest.a $(LDADD)
wget_url_fuzzer_SOURCES = wget_url_fuzzer.c $(MAIN)
wget_url_fuzzer_LDADD = ../src/libunittest.a $(LDADD)
#EXTRA_DIST = $(wildcard *.options) $(wildcard *.dict) \
# $(wildcard *.in) $(wildcard *.repro)
dist-hook:
find $(srcdir) -name '*.options' -exec cp -v '{}' $(distdir) ';'
find $(srcdir) -name '*.dict' -exec cp -v '{}' $(distdir) ';'
# find $(srcdir) -name '*.in' -exec cp -vr '{}' $(distdir) ';'
find $(srcdir) -name '*.repro' -exec cp -vr '{}' $(distdir) ';'
clean-local:
rm -rf *.gc?? *.log lcov
oss-fuzz:
if test "$$OUT" != ""; then \
XLIBS="-lpsl -lgnutls -lnettle -lhogweed -lidn2 -lunistring"; \
for ccfile in wget*_fuzzer.c; do \
fuzzer=$$(basename $$ccfile .c); \
$$CXX $$CXXFLAGS -I$(top_srcdir)/src -I$(top_srcdir) -I$(top_srcdir)/lib \
"$${fuzzer}.c" -o "$${fuzzer}" \
../src/libunittest.a ../lib/libgnu.a $${LIB_FUZZING_ENGINE} \
-Wl,-Bstatic $${XLIBS} -Wl,-Bdynamic; \
done; \
fi
.PHONY: oss-fuzz

84
fuzz/README.md Normal file
View File

@@ -0,0 +1,84 @@
# Fuzzers
These are fuzzers designed for use with `libFuzzer` or `afl`. They can
be used to run on Google's OSS-Fuzz (https://github.com/google/oss-fuzz/).
The convention used here is that the initial values for each parser fuzzer
are taken from the $NAME.in directory.
Crash reproducers from OSS-Fuzz are put into $NAME.repro directory for
regression testing with top dir 'make check' or 'make check-valgrind'.
# Running a fuzzer using clang
Use the following commands on top dir:
```
export CC=clang-6.0
export CXX=clang++-6.0
# address sanitizer:
export CFLAGS="-O1 -g -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=undefined,integer,nullability -fsanitize=address -fsanitize-address-use-after-scope -fsanitize-coverage=trace-pc-guard,trace-cmp"
# undefined sanitizer;
export CFLAGS="-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=bool,array-bounds,float-divide-by-zero,function,integer-divide-by-zero,return,shift,signed-integer-overflow,vla-bound,vptr -fno-sanitize-recover=bool,array-bounds,float-divide-by-zero,function,integer-divide-by-zero,return,shift,signed-integer-overflow,vla-bound,vptr -fsanitize=fuzzer-no-link"
export CXXFLAGS="$CFLAGS -stdlib=libc++"
export LIB_FUZZING_ENGINE="-lFuzzer -lstdc++"
./configure --enable-fuzzing --without-metalink --without-zlib --disable-pcre --without-libuuid
make clean
make -j$(nproc)
cd fuzz
# run wget_options_fuzzer
UBSAN_OPTIONS=print_stacktrace=1 ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-6.0/bin/llvm-symbolizer \
./run-clang.sh wget_options_fuzzer
```
If you see a crash, then a crash corpora is written that can be used for further
investigation. E.g.
```
==2410==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000004e90 at pc 0x00000049cf9c bp 0x7fffb5543f70 sp 0x7fffb5543720
...
Test unit written to ./crash-adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
```
To reproduce the crash:
```
./wget_options_fuzzer < ./crash-adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
```
You can also copy/move that file into wget_options_fuzzer.repro/
and re-build the project without fuzzing for a valgrind run, if you like that better.
Just a `./configure` and a `make check-valgrind` should reproduce it.
# Running a fuzzer using AFL
Use the following commands on top dir:
```
$ export LIB_FUZZING_ENGINE=""
$ CC=afl-clang-fast ./configure --enable-fuzzing
$ make -j$(nproc) clean all
$ cd fuzz
$ ./run-afl.sh wget_options_fuzzer
```
# Fuzz code coverage using the corpus directories *.in/
Code coverage reports currently work best with gcc+lcov+genhtml.
In the top directory:
```
CC=gcc CFLAGS="-O0 -g" ./configure
make fuzz-coverage
xdg-open lcov/index.html
```
To work on corpora for better coverage, `cd fuzz` and use e.g.
`./view-coverage.sh wget_options_fuzzer`.
# Creating wget_options_fuzzer.dict
```
for i in `../src/wget --help|tr ' ' '\n'|grep ^--|cut -c 3-|sort`;do echo \"$i\"; done >wget_options_fuzzer.dict
```

26
fuzz/fuzzer.h Normal file
View File

@@ -0,0 +1,26 @@
/*
* Copyright(c) 2017-2018 Free Software Foundation, Inc.
*
* This file is part of GNU Wget.
*
* GNU Wget is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* (at your option) any later version.
*
* GNU Wget is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with Wget. If not, see <https://www.gnu.org/licenses/>.
*/
#include <stddef.h> // size_t
#include <stdint.h> // uint8_t
#ifdef __cplusplus
extern "C"
#endif
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);

11
fuzz/get_all_corpora Executable file
View File

@@ -0,0 +1,11 @@
#!/bin/sh -eu
if ! grep -q FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION Makefile; then
echo "The fuzzers haven't been built for fuzzing (maybe for regression testing !?)"
echo "Please built regarding README.md and try again."
exit 1
fi
for fuzzer in *_fuzzer.c; do
./get_ossfuzz_corpora $(basename $fuzzer .c)
done

46
fuzz/get_ossfuzz_corpora Executable file
View File

@@ -0,0 +1,46 @@
#!/bin/sh -eu
# As a first step see README.md and follow the steps under "Running a fuzzer using clang".
# You might need 'gsutil' to download new corpora from the Google cloud:
# Read the docs at https://github.com/google/oss-fuzz/blob/master/docs/corpora.md
# then install 'google-cloud-sdk' and execute 'gcloud init'.
# Now 'gsutil' should be ready to use.
if test -z "$1"; then
echo "Usage: $0 <fuzzer target>"
echo "Example: $0 wget_options_fuzzer"
exit 1
fi
if ! grep -q FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION Makefile; then
echo "The fuzzers haven't been built for fuzzing (maybe for regression testing !?)"
echo "Please built regarding README.md and try again."
exit 1
fi
fuzzer=$1
project=wget
# sync/copy the OSS-Fuzz corpora into the .new directory
mkdir -p ${fuzzer}.new
cp -fp ${fuzzer}.in/* ${fuzzer}.new
gsutil cp $(gsutil ls gs://${project}-backup.clusterfuzz-external.appspot.com/corpus/libFuzzer/${fuzzer}|tail -n 1) ${fuzzer}.new
(cd ${fuzzer}.new && unzip -q -f -o *.zip && rm *.zip)
# create fuzzer target
BUILD_ONLY=1 ./run-clang.sh ${fuzzer}
# merge the corpora into the .in directory
./${fuzzer} -merge=1 ${fuzzer}.in ${fuzzer}.new
# now clear .new dir and put all corpora there
rm -rf ${fuzzer}.new
mv ${fuzzer}.in ${fuzzer}.new
mkdir ${fuzzer}.in
# now merge again (optimizes number of corpora)
./${fuzzer} -merge=1 ${fuzzer}.in ${fuzzer}.new
echo
echo "If new files have been added, 'git add' and 'git commit' them."

32
fuzz/glob_crash.c Normal file
View File

@@ -0,0 +1,32 @@
/*
* Created 19.10.2017 by Tim Rühsen
*
* Call glob() using data from fuzzer crash file
*
* Build and execute with instrumented gnulib (amend -I paths as needed):
*
* clang build (spills out WRITE heap buffer overflow)
* export CC=clang-6.0
* export CFLAGS="-O1 -g -fno-omit-frame-pointer -fsanitize=address -fsanitize-address-use-after-scope"
* $CC $CFLAGS -I.. -I../lib glob_crash.c -o glob_crash ../lib/.libs/libgnu.a
* ./glob_crash
*
* gcc build (spills out READ heap buffer overflow):
* export CC=gcc
* export CFLAGS="-O1 -g -fno-omit-frame-pointer -fsanitize=address -fsanitize-address-use-after-scope"
* $CC $CFLAGS -I.. -I../lib glob_crash.c -o glob_crash ../lib/.libs/libgnu.a
* ./glob_crash
*/
#include <glob.h>
int main(int argc, char **argv)
{
static unsigned char data[] = "1e";
glob_t pglob;
if (glob(data, GLOB_TILDE|GLOB_ONLYDIR|GLOB_NOCHECK, NULL, &pglob) == 0)
globfree(&pglob);
return 0;
}

154
fuzz/main.c Normal file
View File

@@ -0,0 +1,154 @@
/*
* Copyright(c) 2017-2018 Free Software Foundation, Inc.
*
* This file is part of GNU Wget.
*
* GNU Wget is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* (at your option) any later version.
*
* GNU Wget is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with Wget. If not, see <https://www.gnu.org/licenses/>.
*/
#include <config.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include <fcntl.h>
#include "fuzzer.h"
#include "utils.h"
#ifdef TEST_RUN
#include <dirent.h>
#ifdef _WIN32
# define SLASH '\\'
#else
# define SLASH '/'
#endif
static int test_all_from(const char *dirname)
{
DIR *dirp;
if ((dirp = opendir(dirname))) {
struct dirent *dp;
while ((dp = readdir(dirp))) {
if (*dp->d_name == '.') continue;
char fname[strlen(dirname) + strlen(dp->d_name) + 2];
snprintf(fname, sizeof(fname), "%s/%s", dirname, dp->d_name);
struct file_memory *fmem;
if ((fmem = wget_read_file(fname))) {
printf("testing %ld bytes from '%s'\n", fmem->length, fname);
LLVMFuzzerTestOneInput((uint8_t *)fmem->content, fmem->length);
wget_read_file_free(fmem);
}
}
closedir(dirp);
return 0;
}
return 1;
}
int main(int argc, char **argv)
{
// if VALGRIND testing is enabled, we have to call ourselves with valgrind checking
const char *valgrind = getenv("VALGRIND_TESTS");
const char *target;
size_t target_len;
if (!valgrind || !*valgrind || !strcmp(valgrind, "0")) {
// fallthrough
}
else if (!strcmp(valgrind, "1")) {
char cmd[strlen(argv[0]) + 256];
snprintf(cmd, sizeof(cmd), "VALGRIND_TESTS=\"\" valgrind --error-exitcode=301 --leak-check=yes --show-reachable=yes --track-origins=yes %s", argv[0]);
return system(cmd) != 0;
} else {
char cmd[strlen(valgrind) + strlen(argv[0]) + 32];
snprintf(cmd, sizeof(cmd), "VALGRIND_TESTS="" %s %s", valgrind, argv[0]);
return system(cmd) != 0;
}
if ((target = strrchr(argv[0], SLASH)))
target = strrchr(target, '/');
else
target = strrchr(argv[0], '/');
target = target ? target + 1 : argv[0];
if (strncmp(target, "lt-", 3) == 0)
target += 3;
target_len = strlen(target);
#ifdef _WIN32
target_len -= 4; // ignore .exe
#endif
{
int rc;
char corporadir[sizeof(SRCDIR) + 1 + target_len + 8];
snprintf(corporadir, sizeof(corporadir), SRCDIR "/%.*s.in", (int) target_len, target);
rc = test_all_from(corporadir);
if (rc)
fprintf(stderr, "Failed to find %s\n", corporadir);
snprintf(corporadir, sizeof(corporadir), SRCDIR "/%.*s.repro", (int) target_len, target);
if (test_all_from(corporadir) && rc)
return 77; // SKIP
}
return 0;
}
#else
#ifndef __AFL_LOOP
static int __AFL_LOOP(int n)
{
static int first = 1;
if (first) {
first = 0;
return n && --n > 0;
}
return 0;
}
#endif
int main(int argc, char **argv)
{
int ret;
unsigned char buf[64 * 1024];
while (__AFL_LOOP(10000)) { // only works with clang - we have to use 1 because static/global vars in wget
ret = fread(buf, 1, sizeof(buf), stdin);
if (ret < 0)
return 0;
LLVMFuzzerTestOneInput(buf, ret);
}
return 0;
}
#endif /* #ifdef TEST_RUN */

59
fuzz/run-afl.sh Executable file
View File

@@ -0,0 +1,59 @@
#!/bin/sh -eu
# Copyright(c) 2017 Free Software Foundation, Inc.
#
# This file is part of GNU Wget.
#
# GNU Wget is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# GNU Wget is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with GNU Wget. If not, see <https://www.gnu.org/licenses/>.
if ! grep -q '^CC=.afl-clang-fast' ../config.log; then
echo "compile first library as:"
echo "CC=afl-clang-fast ./configure"
exit 1
fi
if test -z "$1"; then
echo "Usage: $0 test-case"
echo "Example: $0 wget_options_fuzzer"
exit 1
fi
fuzzer=$1
#rm -f $fuzzer
#afl-clang-fast -O2 -g -I.. main.c "${fuzzer}.c" -o "${fuzzer}"
### minimize test corpora
if test -d ${fuzzer}.in; then
mkdir -p ${fuzzer}.min
for i in `ls ${fuzzer}.in`; do
fin="${fuzzer}.in/$i"
fmin="${fuzzer}.min/$i"
if ! test -e $fmin || test $fin -nt $fmin; then
afl-tmin -m 500 -i $fin -o $fmin -- ./${fuzzer}
fi
done
fi
TMPOUT=${fuzzer}.out
mkdir -p ${TMPOUT}
if test -f ${fuzzer}.dict; then
afl-fuzz -m 500 -i ${fuzzer}.min -o ${TMPOUT} -x ${fuzzer}.dict -- ./${fuzzer}
else
afl-fuzz -m 500 -i ${fuzzer}.min -o ${TMPOUT} -- ./${fuzzer}
fi
echo "output was stored in $TMPOUT"
exit 0

57
fuzz/run-clang.sh Executable file
View File

@@ -0,0 +1,57 @@
#!/bin/bash -e
#
# Copyright(c) 2017 Free Software Foundation, Inc.
#
# This file is part of GNU Wget.
#
# GNU Wget is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# GNU Wget is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with GNU Wget. If not, see <https://www.gnu.org/licenses/>.
trap ctrl_c INT
ctrl_c() {
./${fuzzer} -merge=1 ${fuzzer}.in ${fuzzer}.new
rm -rf ${fuzzer}.new
}
if test -z "$1"; then
echo "Usage: $0 <fuzzer target>"
echo "Example: $0 wget_options_fuzzer"
exit 1
fi
if ! grep -q FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION Makefile; then
echo "The fuzzers haven't been built for fuzzing (maybe for regression testing !?)"
echo "Please built regarding README.md and try again."
exit 1
fi
# you'll need ~2GB free memory per worker !
fuzzer=$1
workers=$(($(nproc) - 0))
jobs=$workers
if test -n "$BUILD_ONLY"; then
exit 0
fi
# create directory for NEW test corpora (covering new areas of code)
mkdir -p ${fuzzer}.new
if test -f ${fuzzer}.dict; then
./${fuzzer} -dict=${fuzzer}.dict ${fuzzer}.new ${fuzzer}.in -jobs=$jobs -workers=$workers
else
./${fuzzer} ${fuzzer}.new ${fuzzer}.in -jobs=$jobs -workers=$workers
fi
exit 0

37
fuzz/view-coverage.sh Executable file
View File

@@ -0,0 +1,37 @@
#!/bin/bash -eu
#
# (C)2017 Tim Ruehsen tim.ruehsen@gmx.de
#
# View the coverage report for one or more fuzzers.
# 1. execute 'make fuzz-coverage' in the top directory
# 2. execute './view-coverage.sh <fuzz target(s)>
# Example with single fuzzer:
# ./view-coverage.sh wget_options_fuzzer
# Example with two fuzzers:
# ./view-coverage.sh wget_options_fuzzer wget_html_parse_fuzzer
if test -z "$1"; then
echo "Usage: $0 <fuzz target(s)>"
echo "Example: $0 wget_options_fuzzer"
exit 1
fi
#fuzzer="./"$1
LCOV_INFO=coverage.info
#./coverage.sh $fuzzer
#lcov --capture --initial --directory ../src/.libs --directory . --output-file $LCOV_INFO
#lcov --capture --directory ../src/.libs --output-file $LCOV_INFO
#lcov --remove $LCOV_INFO '*/test_linking.c' '*/css_tokenizer.lex' '*/<stdout>' '*/*.h' -o $LCOV_INFO
#genhtml --prefix . --ignore-errors source $LCOV_INFO --legend --title "$1" --output-directory=lcov
lcov --zerocounters --directory ../src/
lcov --capture --initial --directory ../src/.libs --directory . --output-file $LCOV_INFO
make check TESTS="$*" CFLAGS="$(CFLAGS) --coverage" LDFLAGS="$(LDFLAGS) --coverage"
lcov --capture --directory ../src/.libs --output-file $LCOV_INFO
lcov --remove $LCOV_INFO '*/css_tokenizer.lex' '*/*.h' -o $LCOV_INFO
genhtml --prefix . --ignore-errors source $LCOV_INFO --legend --title "$*" --output-directory=lcov
xdg-open lcov/index.html

97
fuzz/wget_cookie_fuzzer.c Normal file
View File

@@ -0,0 +1,97 @@
/*
* Copyright(c) 2017-2018 Free Software Foundation, Inc.
*
* This file is part of GNU Wget.
*
* GNU Wget is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* (at your option) any later version.
*
* GNU Wget is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with Wget. If not, see <https://www.gnu.org/licenses/>.
*/
#include <config.h>
#include <sys/types.h>
#include <dirent.h> // opendir, readdir
#include <stdint.h> // uint8_t
#include <stdio.h> // fmemopen
#include <string.h> // strncmp
#include <stdlib.h> // free
#include "wget.h"
#undef fopen_wgetrc
#ifdef __cplusplus
extern "C" {
#endif
#include "cookies.h"
// declarations for wget internal functions
int main_wget(int argc, const char **argv);
void cleanup(void);
FILE *fopen_wget(const char *pathname, const char *mode);
FILE *fopen_wgetrc(const char *pathname, const char *mode);
void exit_wget(int status);
#ifdef __cplusplus
}
#endif
#include "fuzzer.h"
FILE *fopen_wget(const char *pathname, const char *mode)
{
return fopen("/dev/null", mode);
}
FILE *fopen_wgetrc(const char *pathname, const char *mode)
{
return NULL;
}
#ifdef FUZZING
void exit_wget(int status)
{
}
#else
void exit(int status)
{
}
#endif
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
FILE *bak;
struct cookie_jar *cookie_jar;
char *set_cookie;
if (size > 1024) // same as max_len = ... in .options file
return 0;
set_cookie = (char *) malloc(size + 1);
memcpy(set_cookie, data, size);
set_cookie[size] = 0;
bak = stderr;
stderr = fopen("/dev/null", "w");
cookie_jar = cookie_jar_new();
cookie_handle_set_cookie(cookie_jar, "x", 81, "p", set_cookie);
cookie_handle_set_cookie(cookie_jar, "x", 81, "p", set_cookie);
cookie_handle_set_cookie(cookie_jar, "x", 80, "p/d/", set_cookie);
cookie_jar_delete(cookie_jar);
fclose(stderr);
stderr = bak;
free(set_cookie);
return 0;
}

View File

@@ -0,0 +1,6 @@
tag1="expires="
tag2="path="
tag3="domain="
tag4="httponly"
tag5="secure"
tag6="max-age"

View File

@@ -0,0 +1 @@
__Host-ID=x; expires=MoHos__Host-ID=x; expires=Mon, 29 Feb 2016 07:48:97 GMT; mh=/; domain=x.y; httponly; sec__Host-ID=x; expires=Mon, 29 Feb 2016 07:48:97 GMT; mh=/; domain=x.y; htut-ID=x; expires=Mon, 29 Feb 2016 07:48:97 GMT; mh=/; domain=x.y; hture

View File

@@ -0,0 +1 @@
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDNDDDDD廛max-ageDDDDDD'DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD僖DDDDDhttponlyDDDDDDDDDDDDDDDDDDD;

View File

@@ -0,0 +1 @@
__Hos_o=;;yõ=m;tmID=m;=;y=m;y=m;<;yõ=m;tmID=m;

View File

@@ -0,0 +1 @@
__expires=x; expires=Mon, 14 4032 07:48Mon, 14 4032 07:48:5$ GMT; stmID=x; expires=Mon, 14 4000 07:48:54 GMT; matp=x; expires=Mon, 43 4032 07:48:5$ G032 07:48:27 GMT; stm 14 4000 07ÅË8:54 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%GMT; m:48:54 GMT; stmI»D=x; e|pires=Mon, 14 4000 07ID=x; expires=Mon, 07 4032 07:48:5$ GMT; stmID=x; expin, 14 4032 07:48:5$ GMT; stmID=x; expires=Mon, 14 4000 07:48:54 GMT; matp=x; expires=Mon, 14 4032 07:48:5$ G!32 07:48:54 GMT; stm 14 4000 07ÅË8:54 GMT; m:48:54 GMT; stmI»n4 1, 4032 07:48:5$ GMT; stmID=x; expcure

View File

@@ -0,0 +1,2 @@
__H@=x qqqqqqqqqqqqqqqqq*qqqqqqFpath=; secure
x-age; secure

View File

@@ -0,0 +1 @@
_dict=libwget_cookie_fuzzer.dmutate_depth _e=;Ä;Ø;;;;Ø;;;;Ä;=;;Ø;;;;Ä;=;__Host-I@=x; expires=Mon, 29 Feb@=x; expir st-016 07:48es=Mon, 28 Feb 2016 07::54 GMT;"max-48:

View File

@@ -0,0 +1 @@
_use¡<>šÂÊà pa<70>h=_Host-ID=xGMT; max-age=5; pa<70>h=ÑÄß<C39F>ž•n=x.y; httpomax-agecure

View File

@@ -0,0 +1 @@
*=';:n*='d

View File

@@ -0,0 +1 @@
__Host-ID=x; expires=29 Feb 2016 07:48:54 GMT;t-ID=x; expires=29 Feb 2016 10:48:54 GMT; max-age=5; p; expires=29 Feb 2016 07:48:54 GMT;t-ID=x; expires=58 Feb 2016 10:48:54 GMT; max-age=5; pathomt-ID=x; expires=29 Feb 2016 07:48:54 GMT;t-IDBx; exp=x.y; athomt-ID=x; expires=29 Feb 2016 07:48:54 GMT;t-IDBx; expires=29 Feb 2016 07:48:54 GMT; max-age=5; pathomai;ost-ID=x; expires=29 Feb 2016 07:48:54 GMT;t-ID=x; expires=2; Feb 2016 10:48:5: GMT; max-age=5; pathomt-ID=x; expires=29 Feb 2016 07:48:54 GMT;t-ID=x; expires=29 Feb 2096 07:48:54 GMT; max-age=5; pathomai;=x.y; httponly

View File

@@ -0,0 +1 @@
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

View File

@@ -0,0 +1 @@
xexpires= domain==/; domain==/; domain=th=/; domain=th=/; domain=th=/; domain==/; domain=th=/; domain==/; domain=th=/; domain==/; domain=ht=/; domain==/; domain=th=/; domain==/; domain=th=/expires=; domain==/; domain=th=/;omain==/;*=/; >domain=h=/; domain==/; doamin==/; doiain=th=/; domain=th=/:; domain=th=/; domain==/; domain=th=/; dom'ain==/; domain=th=/; domain==/; domain=th=/; ƒƒƒƒƒƒƒdomain==/; domain=th=/; domai=th=/; domain ==/; domain=th=/expires=; domain==/; domain=th=/;omain==/; domain=th=/; domain==/; domain=th=/expires=; domainy=

View File

@@ -0,0 +1 @@
_httpominimize_crash16 07:48??¿??????????????????????????????????,,,,,,,,,,6:48??¿????????????????????????expires=??????????,,,,,,,,,,,,,,ŠŠŠŠŠŠŠŠŠŠŠ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,???????54 GMT; max-age=5; paagominimize_crash16 07:48??¿????????????;?????????????????????,,,,,,,,,,7:48??¿????????????????????????expires=??????????,,,,,,,,,,,,,,ŠŠŠŠŠŠŠŠŠŠŠ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,???????5path=2 GMT; max-age=5; paagŠŠŠŠŠŠŠŠŠŠ,,,,,,,,,,,,,,,,,,,,,,,,,l,,,???????54 GMT; ŠŠŠ,,,,,,,,,,,,,,,,,,,,,,,,,l,,,???????54 GMT;ge=5=x.ÿÿÿ ly; sñcure

View File

@@ -0,0 +1,2 @@
_ÿÿÿÿÿÿ
ä=x; expi!res=-86148:54 =x; expires=-43855 Feb 6119 07:48:0Tx; expires=-437708 Feb 6120 07:48:54 8:54 =x; expires=-86148:54 =x; expires=-43855 Feb 6119 07:48:5Tx; expires=-437708 Feb 6120 07:48:54 8:54 =x; expires=-43855 Feb 6119 07:48:5TxC; expires=-437708 Feb 120 07:48:54 =b 6119 07:48:5Tx; expires=-218854 Feb 6120 07:48:54 =_axost-ID=x; 6120 07:48:54 =st-ID=x; expires=-87708 Feb 6119 07:48:54 =x; expires=3-4855 Feb 6119 07:48:5Tx; emax-age43855 Feb 6119 07:48:5Tx; expires=-437708 Feb 6120 07:48:54 =b 6119 07:48:5Tx; expires=-437708 Feb 6120 07:48:54 48:5Tx; expires=-437708 Feb 6120 07:48:54 =_axost-ID=x; ex88888888888888888888888888888888888888888pires=-c87708 Feb 6119 07:48:54 =x; expires=-43855 Feb 6119 07:48:5Tx; expires=-43855 Feb 611907:48:5T

View File

@@ -0,0 +1 @@
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

View File

@@ -0,0 +1 @@
ě=;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;D;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;mai;Dmaa;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;Dmai;=

View File

@@ -0,0 +1 @@
_ÿÿÿÿÿÿ ¤seedANShttponly.;domain=.;_;D=_;domain=.;domain=.;_;;domain=.;domain=.;;;doma=o;id;n.main=.;;d(o'main=.;domainn=.;domain=.;_;Domain=.;(;domain=.=_;domain=.;domain=.;_;;domain=.;domaiQQQQQQQQQQQQQQQQQQQQQQQQQQQQomain=.;;domain=.;domain=.;domain=.;(;domain=.;;;domain=.;domain=.;;;domain=.;d_D=_;domain=.;domain=.;_;n=.;domain=.;;;domain=.;;domain=.;;domain=.;domainain=.;domain=.;_;;domain=.;domain=.;;;domain=.;;domain=.;;domain=.;domain=.;domain=.;(;domain=.;;;domain=.;domain=.;om

View File

@@ -0,0 +1,3 @@
__Host-ID=x; 4032 age=2;HHHHHHdomain=HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHh=l; domain=x.y; httpohttponly; sñcurnly;ýly; seceur
e
securd

View File

@@ -0,0 +1 @@
:=;domain=';path=/p/

View File

@@ -0,0 +1 @@
_o=;=;=;y=;;=;=;;<3B>y=_e=';˙˙ qa<71>h=_Host-ID=xGMT; <20>h=_Host-ID=xGMT; domain=˙˙˙˙˙˙˙httponly; securSSSSSSSSSSSSSSSSnly; securSSSSSÓSSSSSSSSSSSSSSSSSSSSSSSSSSSS;=;=;=SSSSSSSSSS;;=;=;=;y=;=;=;=;;y=;=;=;=;=;=;=;==;=SSSS;=;==SSSSSSSSSSSSSSSSSSSS:SSSe

View File

@@ -0,0 +1 @@
_D=_;domain=.;domain=.;domain=.;;;do“ž“=.;;domain=.;;domain=.;domTTT(TTTTTTTTTTTTTTT*************************************************************************************TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTainn=.;domainin=.;domain=.;;;doman;=o;i.dmain=.;;domain=.;domain=.;domain=.;(;domain=.;;;domain=.;domain=.;om

View File

@@ -0,0 +1 @@
WWhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

View File

@@ -0,0 +1 @@
__Host-KDFeec)ure

View File

@@ -0,0 +1,2 @@
__Hos__Host-ID__Host-ID=x; expires=Mon, 5=x.y; httpontsecure-IDmain=ontsecureontsecure-ID=x; expires=Mon, 29 Feb 2016UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU«ªªªªªªªUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUax-age=2; pat$=/; x.y; httponly; ly; secure
secure

View File

@@ -0,0 +1,2 @@
__H@=x @=x; expires=Mon, 29 Femmmmmmmmmmmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq*qqqqqqFpath=; secure
x-age; secure

View File

@@ -0,0 +1 @@
__Host-ID=x; expires=29 Feb 2016 07:48:54 GMT;t-ID=x; expiMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM³¨MMMMMMMMMMMMMMMMMMMMMMMMMres=29 Feb 2expires=016 10:48:54 GMT; max-ageGMT; max-age=5; pathomt-ID=x; expires=29 Feb 2016 07:48:54 GMT;t-IDBx; exp=x.y; athomt-ID=x; expires=58 Feb 2016 07:48:54 GMT;t-IDBx; expires=29 Feb 2016 07:48:54 GM;t-IDBx; exp=x.y; athomt-ID=x; expires=29 Fÿÿÿÿÿÿÿÿÿÿeb 2016 07:48:54 GMT;t-IDBx; expires=29 Feb 2016 07:48:54 GMT; max-age=5; pathomai;ost-ID=x; expires=29 FT; max-age=5; pathomai;ost-ID=x; expires=29 Feb 2016 07:48:54 GMT;t-ID=x; expires=2; Feb 2016 10:48:5: GMT; max-age=5; pathomt-ID=x; max-age=5; pathomai;=x.y; httponly

View File

@@ -0,0 +1 @@
__Host-ID=x; e httponjjjjjjjjjjjjjjjjjjjjzjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjzjjjjjjjjjjjjjjjjjjjjjjjjGMT;Žexpires=n=x.; sec}re

View File

@@ -0,0 +1 @@
_D=_;domain=.;domain=.;_;D=_;domain=.;domain=.;_;;domain=.;domain=.;;;domain=.;;domain=.;;domain=.;domainn=.;domain=.;_;Domain=.;(;domain=.=_;domain=.;domain=.;_;;domain=.;domain=.;;;domain=.;;domain=.;;domain=.;domain=.;domain=.;(;domain=.;;;domain=.;domain=.;;;domain=.;d_D=_;domain=.;domain=.;_;D=_;dom_;D=_;domain=.;domain=.;_;;domain=.;domain==_;dom_;D=_;domain=.;domain=.;_;;domain=.;domain=.;;;domain=.;_NNN)NNNNires=M/n,çË4 F; dNNNNNNNNNNNNNNNNy; ;domain=.;;domain=.;domainn=.;domain=.;_;D=_;domain=.;domain=.;domain=.;;;domain=.;domain=.;;;domain=.;;domain=..;;;domain=.;;domain=.;;domain=.;domainn=.;domain=.;_;D=_;domain=.;domain=.;domain=.;;;domain=.;domain=.;;;domain=.;;domain=.;;domain=.;domainhttp=.;domain=.;;domain=.;;;domain=.;domain=.;domain=.;(;domain=.;m.=ndin=.;;;domain=.;d_D=_;domain=.;domain=.;_;D=_;domain=.;domain=.;_;;doin=.;;;domain=.;domain=.;;;domain=.;;domainin=.;domain=.;domain=.;(;domain=.;;;domain=.;domain=onlyre.;om

View File

@@ -0,0 +1,4 @@
(_pathnul)__Hos__Host-Id=x; expires,M 2on=domain=064 07:4:854 st-Id=x; expires,M 2on=domain=064 07:4:854(nuŽ)_ GMT; max-)__Hos__Host-Id=x; expires,M 2on=domain=064 07:4:854 st-Id=x; expires,M 2o)n=domain=064 07:4:854(nuŽ)_ GMT; max-À;e=4 páth:=/; s__Host-Id=x; expires,M 2on=domain=064 07:4:854 st-Id=x; expires,M 2on=domain=064 07:4:854 GMT; a; ÿÿÿ)GMmaire
secÀ;e=4 páth:=/; s__Host-Id=x; expires,M 2on=domain=064 07:4:854 st-Id=x; expires,M 2on=domain=064 07:4:854 GMT; a__Hosnul)__Hos__Host-Id=x; expires,M 2on=domain=064 07:4:854 st-Id=x; expires,M 2on=domain=064 07:4:854(nuŽ)_ GMT; max-)__Hos__Host-Id=x; expires,M 2on=domain=064 07:4:854 st-Id=x; expires,M 2o)n=domain=064 07:4:854(nuŽ)_ GMT; max-À;e=4 páth:=/; s__Host-Id=x; expires,M 2on=domain=064 07:4:854 st-I__Host-Id=x; expires,M 2on=domain=064 07:4:854 st-Id=x; expires,M 2on=domain=064 07:4:854(nuŽ)_ GMT; maire
secure

View File

@@ -0,0 +1 @@
9=;exp=;expires=¹;expires=8=;expires=¹;expires=¹99=;expires=¹;expires==;expires=¹;expires=¹;e;expires=¹;expires=¹99kk=;expires=¹;expires==;expires=¹;expires=¹;expires==ires=¹;expires=8=;expires=¹;expires=¹99=;expires=¹;expires==;expires=¹;expires=¹;e;exp_res=¹;expires=¹;expires==ires=¹;expires=8=;expires=¹;expires=¹99=;expires=¹;expires==;expires=¹;expires=¹;e;exps=¹;expires=8=;expires=¹;expires=¹99=;expires=¹;expires==;expires=¹;expires=¹;e;expires=¹;expires=¹;expires==ires=¹;expires=8=;expires=¹;expires=¹;expires=¹99kk=;expires=¹;ex99kk=;expires=¹;expires==;expires=s=¹;expires=¹99=;expires=¹;expires==;expires=¹;expires=¹;e;expires=¹;expires=¹99kk=;expiresA¹;expires==;expires=¹;expires=¹;expires==ires=¹;expires=8=;expires=¹;expires=¹99=;expires=¹;expires==;expires=¹;expires=¹;e;expires=¹;expires=¹;expires==ires=¹;expires=8=;expires=¹;expires=¹99=;expires=¹;expires==;expires=¹==;expires=¹;expires=¹;expires==ires=¹;expires=8=;99kk=;expires=¹;expires==;expires=¹;expires=¹;exp¹;expires=¹;expires==;;e

View File

@@ -0,0 +1 @@
_16 07:48'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''':53 Gmain=xcrse ;

View File

@@ -0,0 +1 @@
xmth=/;.;secure@=x..;secure@=x..;secure@=x..;secpath=ure@=x;xmth=/;.;secure@=x..;sectre@=x..;secure@=x;xmth=/;.;secure@=x..;secure@=x..;secure@=x;xmth=/;.;secure@=httponlyre@=x..;secure@=x;xmth=/;.;secure@=x..;secure@=x..;secure@=x;xmth=/;.;secure@=x..;secure@=x..;secure@=x;xmth=/;.;ctre@=x..;.;secure@=x..;secure@=x;

View File

@@ -0,0 +1 @@
_ANSI_D=_;domain=.;domain=.;domain=.;;;do“ž“=.;;domain=.;;domain=.;domTTTTTTTTTTTTTTTTTTT*************************************************************************************TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTainn=.;domain=.;_;Domain=.;(;domain=.=_;domain=.;domain=.;_;;domain=.;domain=.;;};domain=.;;domain=.;;domain=.;do=_;domain=.;domain=.;_;D=_;domain=*;domain=.;_;;domain=.;domain=.;;;doman;=o;i.dmain=.;;domain=.;domain=.;domain=.;(;domain=.;;;domain=.;domain=.;om

View File

@@ -0,0 +1 @@
x/ict="" <20><>==

View File

@@ -0,0 +1 @@
__Hos__Hostk__Hos__Host-__Hos__Host-ID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT; íID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT;Host-ID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT__Host-__Hos__Host-ID=x; expires=Mon, 29 Feb 4000 07: íax-ÀpD=x; expires-a-ID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT__Host-__Hos__Host-ID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT; íID=x; expires=Mon, 29 Feb 4000 07:48:5-4 , 29 Feb 4000 07: íax-ÀpD=x; expires-a-ID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT__Host-__Hos__Host-ID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT; íID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT;Host-ID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT__Host-__Hos__Host-ID=x; expires=Mon, 29 Feb 4000 07: íax-ÀpD=GMT;Host-ID=x; expires=Mon, 29 Feb 4000 07:48:5-4 GMT__Host-__Hos__Host-ID=x; expires=Mon, 29 Feb 4000 07: íax-ÀpD=x; expires-age=2; p™ontsecurgÏe-ID=xÏ

View File

@@ -0,0 +1 @@
D=seï;DT=;D=þ;D=;DD=;;D=;DD=;D=;D=;D=;D=þ;D=w;DD=;D=;D=;DD=:;D=;DT=;D=;D=w;domain=D=;D=

View File

@@ -0,0 +1 @@
__Host-ID=x; expires=Mon, 44 F ;httponly domain=x.y; httponlyre

View File

@@ -0,0 +1 @@
xmth=/; 4 ..<2E>;<3B>e<EFBFBD>.;<3B>n<EFBFBD>e<EFBFBD>.;<3B>nly=

View File

@@ -0,0 +1 @@
_dict=wget_cook{{{{{{{{{{{{{{{{{{{{{{{{˙˙˙˙˙˙)˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙;;path=/; domain=x.y; h˙˙˙˙˙˙˙˙˙˙˙˙ttđPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPő˙˙˙˙˙ŕŕsńcure

View File

@@ -0,0 +1,2 @@
__Hires=˙{{{{{{{{{{{{{{{{{{{{{{{{{{{{˙˙˙˙˙˙˙˙˙˙˙˙˙˙ ˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙use_cmp˙˙˙˙˙˙VVVVVVVVVVVVVVVVV˙˙˙˙˙˙˙VVVVVVVVVVVVVVVVVVVVVVonly; sy; ecure
sńcure

View File

@@ -0,0 +1,5 @@
ahttponlyi!=x&y;qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqQqqqqqqqqqqqi!=x&y;qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq path=&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&; secure
x-age; sec29 Feb 2015 07:48:54 GMT; max-domai!=x&y;qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq path=&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&; secure
x-age; secuqqqqq path=&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&; secure
x-age; sec29 Feb 2015 07:48:54 GM!=x&y;qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq path=&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&(&&&&&&&&&&&; secure
x-age; secure

View File

@@ -0,0 +1 @@
__Host-ax-domain=x.y; patI@=\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ax-doma®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®®in=x.y; patMon, ;T; #mŸ‡Ò<C392>žŒn=x.y; path=h path=x.y; pt=ahh path=httponl@=x; expires=Mon, ;T; max-domain=x.y;__Host-I@=x; expires=-domain=x.y; patI@=x; eres=Mon, ;T; #max-domain=x.y; patMon, ;T; #max-domain=x.y;p atx; expires=Mon, ;T; max-doure

View File

@@ -0,0 +1,11 @@
__Hos__Host-ID=x; expires=Mottpont-ID=x; exp7:48:54 GMT; mires=Moe
secure
aø-Àge=5; path=/; domain=x.y exp7:48:57:48:54 GMT; mires=Moe
secure
aø-À&&&&&&&ge=5; path=/; domain=x.y exp7:48:54 GMT; mires=Moe
secuANT; max-\ge=2ÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇ; xain=x.y; hexp4 GMT; mires=Moe
seexpires=Mottpont-ID=x; exp7:48:54 GMT; mires=Moe
secure
aø-Àge=5; path=/; domain=x.y exp7:48:54 GMT; mires=Moe
secuANT; max-\ge=2ÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇ; xaÇÇÇÇÇÇ; xain=x.y; hcuANT; max-\ge=2ÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇ9>; xain=x.y; hexpiÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇsecureÇÇÇÇÇÇÇÇÇÇ; xain=x.y; hexpires= ly; secure
secure

View File

@@ -0,0 +1 @@
Pxdict="get_cookmagth=/doiimain=~"~~~~~~~~~~~~~~~~~max-age~~~~~~~~~~~~~~~~~~~~~~~~~~~~=˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙˙==/;(domain=th=/=/; domaininy=

View File

@@ -0,0 +1,4 @@
__Hos__Host-ID=x; expires=Mon, 29 Feb 4000 07:48:54 GMT; max-Àge=5; path=/; domain=x.y; main=x.y; httpont-ID=x; expires=Mon, 29 Feb 2016 07:48:54 GMT; max-age=2; path=/; domain=x.y; httponly; ly; sCCCCCCCCCCCCCCCCCCCCCCCCCCCe}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}cure__Ho=x; expires=Mon, 29 Feb 4000 0:7:48:54 GMs__Host-ID=x; expires=Mon, 29 Feb 4000 07:48:54 GT; max-Àge=5; path=/; domain=x.y; httpont-Îecureexpires=Mon, 29 Feb 2016 0::48:54MT; max-Àge=5; path=/; domain=x.y; httpont-ID=x; expires=Mon, 29 Feb 2016 07:48:54 GMT; max-age=2; pa GMT; max-age=2; path=/; domai
seth=/; domai
secure
cure

View File

@@ -0,0 +1,2 @@
__Hos__Host-ID__Host-IDcure-ID=x; expires=Mon, 32 Feb 2016UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU«ªªªªªªªUUUUUUUost-IDcure-ID=x; expires=Mon, 32 Feb 2016UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU«ªªªªªªªUUUUUUUUUUUUUUUUUUU¥ªUUUUUUUUUUUUUUax-a httponly; ly; sUUUUUUUUUUUU¥ªUÕUUUUUUUUUUUUax-a httponly; ly; secure
secure

View File

@@ -0,0 +1,2 @@
__Hos_ßH6 07:48:54 ÿÿÿÿ; expires=Modict=wget_cookie_:48:54 ÿÿÿy; èttpont-ID=x; expires=MANSI_on, 29 Fgb 2016 07:4 httponly;GANSI_*MT; max-age=httponlySt0ctypeIcE/; domain=x.y; httponly; ly; sec!re
secure

View File

@@ -0,0 +1 @@
=;*=;*=;*=;*=;*=;*=;*==;*=;*=;*=;*=;*=;*=þ;*=;*=;**=;*=;*=;*=;*=;*=;*==;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;**=;*=;*=;*=;*=;*=;*==;*=;*=;*=;=;*=;*=;*=;*=;*==;*=;*=;*=;*=;*=;*=;*==;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;**=;*=;*=;*=;*=;*=;*==;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;**=;*=;*=;*=;^^*=;*=;*==;*=;*;*=;*=;*=secure;**=;*=;*=;*=;*=;*=;*==;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;**=;*=;*=;*=;*=;*=;*==;*=;*=;*=;=;*=;*=;*=;*=;*=;*=;*==;*=;*=;*=;*=jjj;*=;*=;*=;*=;**=;*=;*=;*=;*=;*=;*==;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;*=;**=;*=;*=;

View File

@@ -0,0 +1 @@
xdicp="get_cookmagth=/doiimain=~"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~=ÿÿÿÿÿÿÿxpi…šŒÂÄß<C39F>mainy=

View File

@@ -0,0 +1 @@
xmagth=/; domain=~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~=/; domain==/; domain=th=/; domain=th=/; domain=th=/; domain==/; domain=th=/; domain==/; domain=th=/; domain==/; domain=th=/; domain==/; domain=th=/; domain==/; domain=th=/expires=; domain==/; domain=th=/;omain==/; domain=th=/; domain==/; d=/; domain==/; domain=th=/; domain*=/; domain=h=/; domain==/; domain==/; domain=th=/; domain=th=/; domain=th=/;á domain==/; domain=th=/; ain=th=/; domain==/; domain=th=/expires=; domainy=

View File

@@ -0,0 +1 @@
__Host- max-age=5; paph=/; domai29! 2016 07:48omai__Host-IDn, 29! 2016 07:48:54 GMT; max-age=5K path=/; max-age=RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR5 dge=5; path´/; d4 GMT; m

View File

@@ -0,0 +1,3 @@
__Hos__Host-ID=x; expires=Mottpont-ID=x; exp7:48:54 GMT; mires=Moe
secure
aø-Àge=5;ax-\ge=2ÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇ; xain=x.y; hexpiÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇ; xain=x.y; hcuANT; max-\ge=2ÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇÇires= ly; secure(secure

View File

@@ -0,0 +1 @@
_==x;max-age=4; max-age=8; max-age=8ag; max-age=9; max-age=8; max-age=8ag; ax-age=8ag; max-age=9; max-age=8;àmax-age=8ag; max-age=9; max-age=83; max-agm=9a-x;eage=8_;max-age=4; max-age=8; max-age=8=ag; max-age=9; max-age=83; max-age=9;max-age=8_; mmax-age=9; max-age=83; max-age=9;max-age=8_;max-age=4; max-age=8; max-__Host-ID=:54 GMT; max-age=5K path=/; max-age=5; path=T; max-age=5K path=/; max-age=5; path=/7:48:54 GMT; max-age=5K path=/age=8=

Some files were not shown because too many files have changed in this diff Show More