Discussion:
[PATCH] Use O_BINARY to detect whether to pass "rb" to popen
(too old to reply)
Filipe Brandenburger
2015-05-21 05:15:46 UTC
Permalink
Get rid of freadonly_mode and fwriteonly_mode which were set from a
define set by a configure AC_RUN_IFELSE check.

Instead, use the previously already defined FOPEN_READ_BINARY and
FOPEN_WRITE_BINARY which are set based on O_BINARY from fcntl.h, which
will be defined to be non-zero on systems where there is a difference
between text and binary files.

Remove the now unnecessary configure check for whether popen works with
binary mode. That particular check required running a program built by
configure, which made it hard or impossible to cross-compile sharutils
without patching the sources.

Tested that it still builds and that the testsuite of `make check`
passes. Checked that `make distcheck` works as expected.

Tested it on a Linux/glibc system and on Windows and cygwin (the closest
to a system where there is a difference between text and binary files,
though cygwin doesn't really seem to do anything with "b" on popen.)

Tested that cross-compiling works on the pristine sources from the dist
tarball without any modifications to the source tree.

Signed-off-by: Filipe Brandenburger <***@google.com>
---
ChangeLog | 7 +++++++
configure.ac.git | 6 ------
src/local.h.git | 10 ----------
src/shar.c | 20 ++++++++++----------
4 files changed, 17 insertions(+), 26 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 043c5abfa5ce..1784941ec6d7 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2015-05-20 Filipe Brandenburger <***@google.com>
+
+ * src/shar.c: Use FOPEN_{READ,WRITE}_BINARY constants derived from
+ O_BINARY instead of f{read,write}only_binary probed by autoconf.
+ * src/local.h.git: Drop the no longer needed f{read,write}only_mode.
+ * configure.ac.git: Drop the check for popen with "rb" mode.
+
2015-01-06 Eric Blake <***@redhat.com>

build: avoid error message failure on 64-bit hosts
diff --git a/configure.ac.git b/configure.ac.git
index 6ab67d9ed2e2..a669db093e67 100644
--- a/configure.ac.git
+++ b/configure.ac.git
@@ -94,12 +94,6 @@ AM_WITH_DMALLOC
CATALOGS="$new_CATALOGS"
fi]

-AC_RUN_IFELSE(
- [AC_LANG_PROGRAM([], [dnl
- FILE * fp = popen ("date", "rb");
- exit (fp == NULL);])],
- [AC_DEFINE([BINARY_MODE_POPEN], [1], [define if popen supports "rb" mode])]
-)
# INVOKE_LIBOPTS_MACROS
INVOKE_LIBOPTS_MACROS

diff --git a/src/local.h.git b/src/local.h.git
index 8f234cbe5d6e..f778c09f773d 100644
--- a/src/local.h.git
+++ b/src/local.h.git
@@ -22,14 +22,4 @@

INCLUDE-LIST

-#ifdef SHAR_C
-# ifdef BINARY_MODE_POPEN
-static char const freadonly_mode[] = "rb";
-static char const fwriteonly_mode[] = "wb";
-# else
-static char const freadonly_mode[] = "r";
-static char const fwriteonly_mode[] = "w";
-# endif
-#endif
-
#endif /* SHARUTILS_LOCAL_H */
diff --git a/src/shar.c b/src/shar.c
index 5507b3b71962..54bb28d9efd3 100644
--- a/src/shar.c
+++ b/src/shar.c
@@ -1170,7 +1170,7 @@ file_needs_encoding (char const * fname)
average file size, even reading the whole file (if it is text) would
usually be faster than invoking 'file'. */

- infp = fopen (fname, freadonly_mode);
+ infp = fopen (fname, FOPEN_READ_BINARY);

if (infp == NULL)
{
@@ -1265,11 +1265,11 @@ encode_file_to_pipe (
{
sprintf (cmdline, cmpr_state->cmpr_cmd_fmt,
cmpr_state->cmpr_level, q_local_name);
- in_fp = popen (cmdline, freadonly_mode);
+ in_fp = popen (cmdline, FOPEN_READ_BINARY);
}
else
{
- in_fp = fopen (local_name, freadonly_mode);
+ in_fp = fopen (local_name, FOPEN_READ_BINARY);
open_fmt = "fopen";
open_txt = local_name;
}
@@ -1277,7 +1277,7 @@ encode_file_to_pipe (
if (in_fp == NULL)
fserr (SHAR_EXIT_FAILED, open_fmt, open_txt);

- out_fp = fdopen (out_fd, fwriteonly_mode);
+ out_fp = fdopen (out_fd, FOPEN_WRITE_BINARY);

fprintf (out_fp, mode_fmt_z, restore_name);

@@ -1321,7 +1321,7 @@ open_encoded_file (char const * local_name, char const * q_local_name,
close (pipex[1]);

{
- FILE * fp = fdopen (pipex[0], freadonly_mode);
+ FILE * fp = fdopen (pipex[0], FOPEN_READ_BINARY);
if (fp == NULL)
fserr (SHAR_EXIT_FAILED, "fdopen", _("pipe fd"));
return fp;
@@ -1402,7 +1402,7 @@ open_encoded_file (char const * local_name,
sprintf (p, uu_cmd_fmt, restore_name);
}

- /* Don't use freadonly_mode because it might be "rb", while we need
+ /* Don't use FOPEN_READ_BINARY because it might be "rb", while we need
text-mode read here, because we will be reading pure text from
uuencode, and we want to drop any CR characters from the CRLF
line endings, when we write the result into the shar. */
@@ -1441,7 +1441,7 @@ open_shar_input (
*file_type_p = _("text");
*file_type_remote_p = SM_type_text;

- infp = fopen (local_name, freadonly_mode);
+ infp = fopen (local_name, FOPEN_READ_BINARY);
if (infp == NULL)
fserr (SHAR_EXIT_FAILED, "fopen", local_name);
*pipe_p = 0;
@@ -1822,7 +1822,7 @@ finish_sharing_file (const char * lname, const char * lname_q,
echo_status ("test $? -ne 0", SM_restore_failed, NULL, rname, 0);

if ( ! HAVE_OPT(NO_MD5_DIGEST)
- && (fp = fopen (lname, freadonly_mode)) != NULL
+ && (fp = fopen (lname, FOPEN_READ_BINARY)) != NULL
&& md5_stream (fp, md5buffer) == 0)
{
/* Validate the transferred file using 'md5sum' command. */
@@ -2050,7 +2050,7 @@ open_output (void)
if (output_filename == NULL)
parse_output_base_name (OPT_ARG(OUTPUT_PREFIX));
sprintf (output_filename, OPT_ARG(OUTPUT_PREFIX), ++part_number);
- output = fopen (output_filename, fwriteonly_mode);
+ output = fopen (output_filename, FOPEN_WRITE_BINARY);

if (output == NULL)
fserr (SHAR_EXIT_FAILED, _("Opening"), output_filename);
@@ -2152,7 +2152,7 @@ configure_shar (int * argc_p, char *** argv_p)
* Redirect stderr to /dev/null in that case so that
* the results are not cluttered with chatter.
*/
- FILE * fp = freopen ("/dev/null", fwriteonly_mode, stderr);
+ FILE * fp = freopen ("/dev/null", FOPEN_WRITE_BINARY, stderr);
if (fp != stderr)
error (SHAR_EXIT_FAILED, errno,
_("reopening stderr to /dev/null"));
--
2.2.0.rc0.207.ga3a616c
Eli Zaretskii
2015-05-21 16:19:25 UTC
Permalink
Date: Wed, 20 May 2015 22:15:46 -0700
Get rid of freadonly_mode and fwriteonly_mode which were set from a
define set by a configure AC_RUN_IFELSE check.
Instead, use the previously already defined FOPEN_READ_BINARY and
FOPEN_WRITE_BINARY which are set based on O_BINARY from fcntl.h, which
will be defined to be non-zero on systems where there is a difference
between text and binary files.
Remove the now unnecessary configure check for whether popen works with
binary mode. That particular check required running a program built by
configure, which made it hard or impossible to cross-compile sharutils
without patching the sources.
I don't really care, but isn't that backwards? I always thought GNU
projects preferred configure tests to manual #ifdef's based on subtle
attributes of the underlying platforms (such as the fact that O_BINARY
is non-zero).

Maybe I'm missing something, but what is the rationale for these
changes? IOW, what was wrong with the original code? (Apologies if I
missed some discussion where this was explained.)

(One reason why you may wish to leave that configure test is that many
modern platforms support "rb" and "wb" in calls to 'fopen', but not in
calls to 'popen'. AFAIR, glibc doesn't support binary modes in
'popen'.)
Filipe Brandenburger
2015-05-21 16:36:18 UTC
Permalink
Hi Eli,
Post by Eli Zaretskii
Post by Filipe Brandenburger
Remove the now unnecessary configure check for whether popen works with
binary mode. That particular check required running a program built by
configure, which made it hard or impossible to cross-compile sharutils
without patching the sources.
I don't really care, but isn't that backwards? I always thought GNU
projects preferred configure tests to manual #ifdef's based on subtle
attributes of the underlying platforms (such as the fact that O_BINARY
is non-zero).
I think checking for whether popen accepts "rb" / "wb" is fine, but if
it's done, I think it should be done in gnulib and then gnulib could
provide one that accepts it regardless.

However, this was attempted before and rejected because it would
consider glibc as "broken" since it's actually the only one that
doesn't accept it...

Some reference to the original thread:
https://lists.gnu.org/archive/html/bug-gnulib/2010-02/msg00187.html
Post by Eli Zaretskii
Maybe I'm missing something, but what is the rationale for these
changes? IOW, what was wrong with the original code? (Apologies if I
missed some discussion where this was explained.)
The problem is that it's currently impossible to cross-compile
sharutils without butchering the build system.

For instance, this is what Gentoo does to the "configure" script after
it's generated:
https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-arch/sharutils/files/sharutils-4.14-popen-rb.patch?view=markup
Post by Eli Zaretskii
(One reason why you may wish to leave that configure test is that many
modern platforms support "rb" and "wb" in calls to 'fopen', but not in
calls to 'popen'. AFAIR, glibc doesn't support binary modes in
'popen'.)
I think *all* of them support "rb" and "wb" to fopen, isn't that part
of POSIX anyways?

Any suggestions of what's the best way to solve this?

I had 3 suggestions on my original e-mail (which unfortunately wasn't
cc'd to bug-gnu-utils) but you can see them on Bruce's first reply
cc'ing the list:
http://lists.gnu.org/archive/html/bug-gnu-utils/2015-05/msg00003.html

Do you like any of those three?

Cheers,
Filipe
Paul Eggert
2015-05-21 16:51:42 UTC
Permalink
Post by Filipe Brandenburger
Any suggestions of what's the best way to solve this?
I suggest doing what Emacs does. Put this in a header somewhere:

#if O_BINARY
# define FOPEN_BINARY "b"
# define FOPEN_TEXT "t"
#else
# define FOPEN_BINARY ""
# define FOPEN_TEXT ""
#endif

and then call 'popen (cmd, "r" FOPEN_BINARY)'. Emacs does the same
thing with fopen of course. In practice this is good enough.
Filipe Brandenburger
2015-05-21 16:54:59 UTC
Permalink
Hi Paul,
Post by Paul Eggert
Post by Filipe Brandenburger
Any suggestions of what's the best way to solve this?
#if O_BINARY
# define FOPEN_BINARY "b"
# define FOPEN_TEXT "t"
#else
# define FOPEN_BINARY ""
# define FOPEN_TEXT ""
#endif
and then call 'popen (cmd, "r" FOPEN_BINARY)'. Emacs does the same thing
with fopen of course. In practice this is good enough.
sharutils sources already carry a similar construct:

http://git.savannah.gnu.org/cgit/sharutils.git/tree/lib/system.h#n102

#if ! defined(O_BINARY) || (O_BINARY == 0)
# define FOPEN_READ_BINARY "r"
# define FOPEN_WRITE_BINARY "w"
#else
# define FOPEN_READ_BINARY "rb"
# define FOPEN_WRITE_BINARY "wb"
#endif

My latest patch proposes using this one instead of the one detected
from autoconf:

http://lists.gnu.org/archive/html/bug-gnu-utils/2015-05/msg00009.html

I wouldn't mind following up (or replacing the current one) with a
patch that uses FOPEN_BINARY/FOPEN_TEXT same as Emacs does and you
suggested.

Cheers,
Filipe
Eli Zaretskii
2015-05-21 17:09:43 UTC
Permalink
Date: Thu, 21 May 2015 09:36:18 -0700
I think checking for whether popen accepts "rb" / "wb" is fine, but if
it's done, I think it should be done in gnulib and then gnulib could
provide one that accepts it regardless.
However, this was attempted before and rejected because it would
consider glibc as "broken" since it's actually the only one that
doesn't accept it...
Ahm...
The problem is that it's currently impossible to cross-compile
sharutils without butchering the build system.
For instance, this is what Gentoo does to the "configure" script after
https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-arch/sharutils/files/sharutils-4.14-popen-rb.patch?view=markup
AFAIU, you are saying that the popen-binary test relies on running a
program, which of course is a cross-compilation killer.
Post by Eli Zaretskii
(One reason why you may wish to leave that configure test is that many
modern platforms support "rb" and "wb" in calls to 'fopen', but not in
calls to 'popen'. AFAIR, glibc doesn't support binary modes in
'popen'.)
I think *all* of them support "rb" and "wb" to fopen, isn't that part
of POSIX anyways?
I think so, yes. But not for 'popen', AFAIK.
Any suggestions of what's the best way to solve this?
If a better test for popen-binary cannot be written that only compiles
and links a program, then I guess your solution is the next best,
indeed.

Thanks.
Bruce Korb
2015-05-22 00:51:44 UTC
Permalink
OK, I think this will work.
Filipe Brandenburger
2015-05-22 04:38:51 UTC
Permalink
Hi Bruce,
OK, I think this will work. [9999-current.diff]
Yes that looks good!

While at it, you might want to get rid of FOPEN_READ_BINARY and
FOPEN_WRITE_BINARY from here:
http://git.savannah.gnu.org/cgit/sharutils.git/tree/lib/system.h#n102

And replace their use with "r" FOPEN_BINARY and "w" FOPEN_BINARY, I
think there are only two uses of each.

Cheers!
Filipe
Eli Zaretskii
2015-05-22 07:21:49 UTC
Permalink
Date: Thu, 21 May 2015 17:51:44 -0700
OK, I think this will work.
LGTM, although the second changeset looks unrelated (?).

Thanks.
Andreas Schwab
2015-05-22 09:15:22 UTC
Permalink
@@ -1402,12 +1402,12 @@ open_encoded_file (char const * local_name,
sprintf (p, uu_cmd_fmt, restore_name);
}
- /* Don't use freadonly_mode because it might be "rb", while we need
+ /* Don't use "r" FOPEN_BINARY mode because it might be "rb", while we need
text-mode read here, because we will be reading pure text from
uuencode, and we want to drop any CR characters from the CRLF
line endings, when we write the result into the shar. */
{
- FILE * in_fp = popen (cmdline, "r");
+ FILE * in_fp = popen (cmdline, "r" TEXT_MODE);
Didn't you mean FOPEN_TEXT?

Andreas.
--
Andreas Schwab, ***@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
Bruce Korb
2015-05-22 15:32:30 UTC
Permalink
Post by Andreas Schwab
{
- FILE * in_fp = popen (cmdline, "r");
+ FILE * in_fp = popen (cmdline, "r" TEXT_MODE);
Didn't you mean FOPEN_TEXT?
Indeed, and yet it built and went through "make distcheck".
And, yes, the patch included a non-related issue. Someone came up
with a tortured uu-encoded file that would cause uudecode to seg fault.
The patch was against the v4.15.1 tag.

Thank you all!

Regards, Bruce
Filipe Brandenburger
2015-05-22 15:43:39 UTC
Permalink
Hi Bruce,
Post by Bruce Korb
Post by Andreas Schwab
{
- FILE * in_fp = popen (cmdline, "r");
+ FILE * in_fp = popen (cmdline, "r" TEXT_MODE);
Didn't you mean FOPEN_TEXT?
Indeed, and yet it built and went through "make distcheck".
That code was under a !HAVE_WORKING_FORK ifdef so it wasn't being
built... I reproduced the build failure on Linux with ./configure
ac_cv_func_fork_works=no instead.

I also think we should get rid of FOPEN_{READ,WRITE}_BINARY and put
these defines in lib/system.h instead of src/local.h.git.

I'll send you an updated patch with that.
Post by Bruce Korb
And, yes, the patch included a non-related issue. Someone came up
with a tortured uu-encoded file that would cause uudecode to seg fault.
The patch was against the v4.15.1 tag.
Yeah in that case they should probably be separate patches. I'll send
you a sequence you can grab with "git am". I'm almost done testing
them, so expect them shortly.

Cheers,
Filipe
Filipe Brandenburger
2015-05-22 15:45:01 UTC
Permalink
Post by Bruce Korb
And, yes, the patch included a non-related issue. Someone came up
with a tortured uu-encoded file that would cause uudecode to seg fault.
Would be nice to add a test case for it... Is the uu-encoded file that
caused the crash available somewhere public? Or would it be easy to
create a simple test case with the same kind of problem?

Cheers,
Filipe
Bruce Korb
2015-05-22 16:48:27 UTC
Permalink
Post by Filipe Brandenburger
Yeah in that case they should probably be separate patches. I'll send
you a sequence you can grab with "git am". I'm almost done testing
them, so expect them shortly.
I guess I'm lazy. They "should be", but....
Post by Filipe Brandenburger
Would be nice to add a test case for it... Is the uu-encoded file that
caused the crash available somewhere public? Or would it be easy to
create a simple test case with the same kind of problem?
My philosophy on that is that such a test case (which I did use to verify
the patch) would never fire again. It is sufficiently non-obvious that
the likelihood of such a test catching an inadvertent problem is zero.
In other words, if there is or will be another "begin" line that can
cause a fault, it would likely not be caught by such a test. It would
likely need another failure pattern. WRT that, though, some negative
testing would be a good thing -- were this a more actively used product.
Nearly everyone uses MIME-encoding for shipping around files nowadays.
This is a pretty anachronistic project.

Short form: be my guest. I'd happily add them to the suite (of 3).
Bruce Korb
2015-05-23 15:35:26 UTC
Permalink
Post by Filipe Brandenburger
Post by Bruce Korb
And, yes, the patch included a non-related issue. Someone came up
with a tortured uu-encoded file that would cause uudecode to seg fault.
Would be nice to add a test case for it... Is the uu-encoded file that
caused the crash available somewhere public? Or would it be easy to
create a simple test case with the same kind of problem?
$ od -c stackoverflow.uu
0000000 b e g i n 0 ~
0000007
$ uudecode stackoverflow.uu
stackoverflow.uu: Invalid or missing 'begin' line
Reading through the various docs I found lots of weasel words saying
decode is not robust against pathological inputs. Still, the above
is now fixed:

http://autogen.sourceforge.net/data/sharutils-4.15.1.4-ed86.tar.xz

but that is not to say you couldn't cook up another. It's a bit harder now.
There is also a bit more to play with. POSIX currently defines two "begin" lines:

begin-base64 <mode> <decoded-path-name>
begin <mode> <decoded-path-name>

In order to allow arbitrarily international "decoded path name"s, the
current code (as a POSIX extension) added another "-encoded" option
to the begin. It may appear by itself, or also before or after the "-base64",
yielding 5 variations on "begin":

begin-encoded <mode> <encoded-decoded-path-name>
begin-base64-encoded <mode> <encoded-decoded-path-name>
begin-encoded-base64 <mode> <encoded-decoded-path-name>

with the last two being effectively the same. The "encoded-decoded-
Post by Filipe Brandenburger
$ f=stackoverflow.uu ; uuencode -m -e $f < $f
begin-base64-encoded 644 c3RhY2tvdmVyZmxvdy51dQ==
YmVnaW4wfg==
====
the last three lines could be reconstructed into the problematical file.
Filipe Brandenburger
2015-05-29 21:19:33 UTC
Permalink
Hi Bruce!

Just curious if you're planning to release a new sharutils package
containing this patch...

I saw you committed a NEWS entry with 4.15.2:
http://git.savannah.gnu.org/cgit/sharutils.git/commit/?id=04b4f557b5bc0e8864f0246c72810543d5d82f51

On the other hand, there's no v4.15.2 git tag and no new package in the FTP:
ftp://ftp.gnu.org/gnu/sharutils/

No rush... Just want to check whether that's in the plans and might
have just been overlooked...

Cheers!
Filipe
Bruce Korb
2015-05-29 23:45:42 UTC
Permalink
Post by Filipe Brandenburger
Hi Bruce!
Just curious if you're planning to release a new sharutils package
containing this patch...
http://git.savannah.gnu.org/cgit/sharutils.git/commit/?id=04b4f557b5bc0e8864f0246c72810543d5d82f51
ftp://ftp.gnu.org/gnu/sharutils/
No rush... Just want to check whether that's in the plans and might
have just been overlooked...
Not overlooked, just not "high priority". :)
The problem is well understood and the affected parties have workarounds, so nobody is in a box.
"soon". Thanks!

Loading...