UTF-8 corruption bug with diff -y

Sjur Nørstebø Moshagen

2018-11-08 08:47:57 UTC

Hello

Using diff on text files with long lines risk corrupting UTF-8 enocded files when used with the default column width of 130 columns, if a multibyte char happens to be on the border of that limit. The diff command will truncate the resulting diff output in the middle of the byte sequence, producing malformed UTF-8 text.

To reproduce:

diff -y Input-text-1.txt Input-text-2.txt

The bug can be circumvented by setting the column width to a randomly high number, as long as it is higher than the longest diff line produced:

diff -y -W 200 Input-text-1.txt Input-text-2.txt

The files Input-text-1.txt and Input-text-2.txt (UTF-8 encoded) are attached. The text (excluding --------) is also reproduced below in case the attachments are removed during e-mail transfer.

Regards,
Sjur Moshagen

Input-text-1.txt:
--------
"<ja>"
"ja" CC
"<ieÅ¡guÄet>"
"ieÅ¡guhtet" Pron Indef Acc
"ieÅ¡guhtet" Pron Indef Attr
"ieÅ¡guhtet" Pron Indef Gen
"<lÃ¡gan>"
"lÃ¡ga" N Sem/Dummytag Ess
"lÃ¡ga" N Sem/Dummytag Sg Loc South Err/Orth
"lÃ¡gan" A Sem/Hum Attr
"lÃ¡gan" A Sem/Hum Sg Acc Err/Orth-nom-acc
"lÃ¡gan" A Sem/Hum Sg Gen Err/Orth-nom-gen
"lÃ¡gan" A Sem/Hum Sg Nom
"lÃ¡hka" N Sem/Rule Sg Loc South Err/Orth
"<borramuÅ¡at>"
"borramuÅ¡" N Sem/Food Pl Nom
"borramuÅ¡" N Sem/Food Sg Acc PxSg2
"borramuÅ¡" N Sem/Food Sg Gen PxSg2
"borrat" Ex/V TV Der/muÅ¡ N Pl Nom
--------

Input-text-2.txt
--------
"<ja>"
"ja" CC
"<ieÅ¡guÄet lÃ¡gan>"
"ieÅ¡guÄetlÃ¡gan" A Sem/Dummytag Attr Err/SpaceCmpÃ¡gan
"ieÅ¡guÄetlÃ¡gan" A Sem/Dummytag Sg Acc Err/Orthacc Err/SpaceCmpÃ¡gan
"ieÅ¡guÄetlÃ¡gan" A Sem/Dummytag Sg Gen Err/Orthgen Err/SpaceCmpÃ¡gan
"ieÅ¡guÄetlÃ¡gan" A Sem/Dummytag Sg Nom Err/SpaceCmpÃ¡gan
"<borramuÅ¡at>"
"borramuÅ¡" N Sem/Food Pl Nom
"borramuÅ¡" N Sem/Food Sg Acc PxSg2
"borramuÅ¡" N Sem/Food Sg Gen PxSg2
"borrat" Ex/V TV Der/muÅ¡ N Pl Nom
--------