The backport of the PR108900 fix to 14 branch broke building chromium
because static_assert (__LINE__ == expected_line_number, ""); now triggers
as the __LINE__ values are off by one.
This isn't the case on the trunk and 15 branch because we've switched
to 64-bit location_t and so one actually needs far longer header files
to trigger it.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120061#c11
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120061#c12
contain (large) testcases in patch form which show on the 14 branch
that the first one used to fail before the PR108900 backport and now
works correctly, while the second one attempts to match the chromium
behavior and it used to pass before the PR108900 backport and now it
FAILs.
The two testcases show rare problematic cases, because
do_include_common -> parse_include -> check_eol -> check_eol_1 ->
cpp_get_token_1 -> _cpp_lex_token -> _cpp_lex_direct -> linemap_line_start
triggers there
      /* Allocate the new line_map.  However, if the current map only has a
         single line we can sometimes just increase its column_bits instead. */
      if (line_delta < 0
          || last_line != ORDINARY_MAP_STARTING_LINE_NUMBER (map)
          || SOURCE_COLUMN (map, highest) >= (1U << (column_bits - range_bits))
          || ( /* We can't reuse the map if the line offset is sufficiently
                  large to cause overflow when computing location_t values.  */
              (to_line - ORDINARY_MAP_STARTING_LINE_NUMBER (map))
              >= (((uint64_t) 1)
                  << (CHAR_BIT * sizeof (linenum_type) - column_bits)))
          || range_bits < map->m_range_bits)
        map = linemap_check_ordinary
                (const_cast <line_map *>
                  (linemap_add (set, LC_RENAME,
                                ORDINARY_MAP_IN_SYSTEM_HEADER_P (map),
                                ORDINARY_MAP_FILE_NAME (map),
                                to_line)));
and so creates a new ordinary map on the line right after the
(problematic) #include line.
Now, in the spot that r14-11679-g8a884140c2bcb7 patched,
pfile->line_table->highest_location in all 3 tests (also
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120061#c13
) is before the decrement the start of the line after the #include line and so
the decrement is really desirable in that case to put highest_location
somewhere on the line where the #include actually is.
But at the same time it is also undesirable, because if we do decrement it,
then linemap_add LC_ENTER called from _cpp_do_file_change will then
  /* Generate a start_location above the current highest_location.
     If possible, make the low range bits be zero.  */
  location_t start_location = set->highest_location + 1;
  unsigned range_bits = 0;
  if (start_location < LINE_MAP_MAX_LOCATION_WITH_COLS)
    range_bits = set->default_range_bits;
  start_location += (1 << range_bits) - 1;
  start_location &=  ~((1 << range_bits) - 1);

  linemap_assert (!LINEMAPS_ORDINARY_USED (set)
                  || (start_location
                      >= MAP_START_LOCATION (LINEMAPS_LAST_ORDINARY_MAP (set))));
and we can end up with the new LC_ENTER ordinary map having the same
start_location as the preceding LC_RENAME one.
Next thing that happens is computation of included_from:
  if (reason == LC_ENTER)
    {
      if (set->depth == 0)
        map->included_from = 0;
      else
        /* The location of the end of the just-closed map.  */
        map->included_from
          = (((map[0].start_location - 1 - map[-1].start_location)
              & ~((1 << map[-1].m_column_and_range_bits) - 1))
             + map[-1].start_location);
The normal case (e.g. with the testcase included at the start of this comment) is
that map[-1] starts somewhere earlier and so map->included_from computation above
nicely computes location_t which expands to the start of the #include line.
With r14-11679 reverted, for #c11 as well as #c12
map[0].start_location == map[-1].start_location above, and so it is
((location_t) -1 & ~((1 << map[-1].m_column_and_range_bits) - 1)))
+ map[-1].start_location,
which happens to be start of the #include line.
For #c11 map[0].start_location is 0x500003a0 and map[-1] has
m_column_and_range_bits 7 and map[-2] has m_column_and_range_bits 12 and
map[0].included_from is set to 0x50000320.
For #c12 map[0].start_location is 0x606c0402 and map[-2].start_location is
0x606c0400 and m_column_and_range_bits is 0 for all 3 maps.
map[0].included_from is set to 0x606c0401.
The last important part is again in linemap_add when doing LC_LEAVE:
      /* (MAP - 1) points to the map we are leaving. The
         map from which (MAP - 1) got included should be the map
         that comes right before MAP in the same file.  */
      from = linemap_included_from_linemap (set, map - 1);

      /* A TO_FILE of NULL is special - we use the natural values.  */
      if (to_file == NULL)
        {
          to_file = ORDINARY_MAP_FILE_NAME (from);
          to_line = SOURCE_LINE (from, from[1].start_location);
          sysp = ORDINARY_MAP_IN_SYSTEM_HEADER_P (from);
        }
Here it wants to compute the right to_line which ought to be the line after
the #include directive.
On the #c11 testcase that doesn't work correctly though, because
map[-1].included_from is 0x50000320, from[0] for that is LC_ENTER with
start_location 0x4080 and m_column_and_range_bits 12 but note that we've
earlier computed map[-1].start_location + (-1 & 0xffffff80) and so only
decreased by 7 bits, so to_line is still on the line with #include and not
after it.  In the #c12 that doesn't happen, all the ordinary maps involved
there had 0 m_column_and_range_bits and so this computes correct line.

Below is a fix for the trunk including testcases using the
location_overflow_plugin hack to simulate the bugs without needing huge
files (in the 14 case it is just 330KB and almost 10MB, but in the 15
case it would need to be far bigger).
The pre- r15-9018 trunk has
FAIL: gcc.dg/plugin/location-overflow-test-pr116047.c -fplugin=./location_overflow_plugin.so  scan-file static_assert[^\n\r]*6[^\n\r]*== 6
and current trunk
FAIL: gcc.dg/plugin/location-overflow-test-pr116047.c -fplugin=./location_overflow_plugin.so  scan-file static_assert[^\n\r]*6[^\n\r]*== 6
FAIL: gcc.dg/plugin/location-overflow-test-pr120061.c -fplugin=./location_overflow_plugin.so  scan-file static_assert[^\n\r]*5[^\n\r]*== 5
and with the patch everything PASSes.
I'll post afterwards a 14 version of the patch.

The patch reverts the r15-9018 change, because it is incorrect,
we really need to decrement it even when crossing ordinary map
boundaries, so that the location is not on the line after the #include
line but somewhere on the #include line.  It also patches two spots
in linemap_add mentioned above to make sure we get correct locations
both in the included_from location_t when doing LC_ENTER (second
line-map.cc hunk) and when doing LC_LEAVE to compute the right to_line
(first line-map.cc hunk), both in presence of an added LC_RENAME
with the same start_location as the following LC_ENTER (i.e. the
problematic cases).
The LC_ENTER hunk is mostly to ensure included_form location_t is
at the start of the #include line (column 0), without it we can
decrease include_from not enough and end up at some random column
in the middle of the line, because it is masking away
map[-1].m_column_and_range_bits bits even when in the end the resulting
include_from location_t will be found in map[-2] map with perhaps
different m_column_and_range_bits.  That alone doesn't fix the bug
though.
The more important is the LC_LEAVE hunk and the problem there is
caused by linemap_line_start not actually doing
    r = set->highest_line + (line_delta << map->m_column_and_range_bits);
when adding a new map (the LC_RENAME one because we need to switch to
different number of directly encoded ranges, or columns, etc.).
So, in the original PR108900 case that
  to_line = SOURCE_LINE (from, from[1].start_location);
doesn't do the right thing, from there is the last < 0x50000000 map
with m_column_and_range_bits 12, from[1] is the first one above it
and map[-1].included_from is the correct location of column 0 on
the #include line, but as the new LC_RENAME map has been created without
actually increasing highest_location to be on the new line (we've just
set to_line of the new LC_RENAME map to the correct line),
  to_line = SOURCE_LINE (from, from[1].start_location);
stays on the same source line.  I've tried to just replace that with
  to_line = SOURCE_LINE (from, linemap_included_from (map - 1)) + 1;
i.e. just find out the #include line from map[-1].included_from and
add 1 to it, unfortunately that breaks the
c-c++-common/cpp/line-4.c
test where we expect to stay on the same 0 line for LC_LEAVE from
<command line> and gcc.dg/cpp/trad/Wunused.c, gcc.dg/cpp/trad/builtins.c
and c-c++-common/analyzer/named-constants-via-macros-traditional.c tests
all with -traditional-cpp preprocessing where to_line is also off-by-one
from the expected one.
So, this patch instead conditionalizes it, uses the
  to_line = SOURCE_LINE (from, linemap_included_from (map - 1)) + 1;
way only if from[1] is a LC_RENAME map (rather than the usual
LC_ENTER one), that should limit it to the problematic cases of when
parse_include peeked after EOL and had to create LC_RENAME map with
the same start_location as the LC_ENTER after it.

Some further justification for the LC_ENTER hunk, using the
https://gcc.gnu.org/pipermail/gcc-patches/2025-May/682774.html testcase
(old is 14 before r14-11679, vanilla current 14 and new with the 14 patch)
I get
$ /usr/src/gcc-14/obj/gcc/cc1.old -quiet -std=c23 pr116047.c -nostdinc
In file included from pr116047-1.h:327677:21,
                 from pr116047.c:4:
pr116047-2.h:1:1: error: unknown type name ‘a’
    1 | a b c;
      | ^
pr116047-2.h:1:5: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘c’
    1 | a b c;
      |     ^
pr116047-1.h:327677:1: error: static assertion failed: ""
327677 | #include "pr116047-2.h"
       | ^~~~~~~~~~~~~
$ /usr/src/gcc-14/obj/gcc/cc1.vanilla -quiet -std=c23 pr116047.c -nostdinc
In file included from pr116047-1.h:327678,
                 from pr116047.c:4:
pr116047-2.h:1:1: error: unknown type name ‘a’
    1 | a b c;
      | ^
pr116047-2.h:1:5: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘c’
    1 | a b c;
      |     ^
$ /usr/src/gcc-14/obj/gcc/cc1.new -quiet -std=c23 pr116047.c -nostdinc
In file included from pr116047-1.h:327677,
                 from pr116047.c:4:
pr116047-2.h:1:1: error: unknown type name ‘a’
    1 | a b c;
      | ^
pr116047-2.h:1:5: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘c’
    1 | a b c;
      |     ^

pr116047-1.h has on lines 327677+327678:
 #include "pr116047-2.h"
 static_assert (__LINE__ == 327678, "");
so the static_assert failure is something that was dealt mainly in the
LC_LEAVE hunk and files.cc reversion, but please have a look at the
In file included from lines.
14.2 emits correct line (#include "pr116047-2.h" is indeed on line
327677) but some random column in there (which is not normally printed
for smaller headers; 21 is the . before extension in the filename).
Current trunk emits incorrect line (327678 instead of 327677, clearly
it didn't decrement).
And the patched compiler emits the right line with no column, as would
be printed if I remove e.g. 300000 newlines from the file.

2025-05-07  Jakub Jelinek  <jakub@redhat.com>

	PR preprocessor/108900
	PR preprocessor/116047
	PR preprocessor/120061
	* files.cc (_cpp_stack_file): Revert 2025-03-28 change.
	* line-map.cc (linemap_add): Use
	SOURCE_LINE (from, linemap_included_from (map - 1)) + 1; instead of
	SOURCE_LINE (from, from[1].start_location); to compute to_line
	for LC_LEAVE.  For LC_ENTER included_from computation, look at
	map[-2] or even lower if map[-1] has the same start_location as
	map[0].

	* gcc.dg/plugin/plugin.exp: Add location-overflow-test-pr116047.c
	and location-overflow-test-pr120061.c.
	* gcc.dg/plugin/location_overflow_plugin.cc (plugin_init): Don't error
	on unknown values, instead just break.  Handle 0x4fHHHHHH arguments
	differently.
	* gcc.dg/plugin/location-overflow-test-pr116047.c: New test.
	* gcc.dg/plugin/location-overflow-test-pr116047-1.h: New test.
	* gcc.dg/plugin/location-overflow-test-pr116047-2.h: New test.
	* gcc.dg/plugin/location-overflow-test-pr120061.c: New test.
	* gcc.dg/plugin/location-overflow-test-pr120061-1.h: New test.
	* gcc.dg/plugin/location-overflow-test-pr120061-2.h: New test.

This patch addresses an issue in the C preprocessor where incorrect
line number information is generated when processing files with a
large number of lines. The problem arises from improper handling
of location intervals in the line map, particularly when locations
exceed LINE_MAP_MAX_LOCATION_WITH_PACKED_RANGES.

By ensuring that the highest location is not decremented if it
would move to a different ordinary map, this fix resolves
the line number discrepancies observed in certain test cases.
This change improves the accuracy of line number reporting, benefiting
users relying on precise code coverage and debugging information.

libcpp/ChangeLog:

	PR preprocessor/108900
	* files.cc (_cpp_stack_file): Do not decrement highest_location
	across distinct maps.

Signed-off-by: Jeremy Bettis <jbettis@google.com>
Signed-off-by: Yash Shinde <Yash.Shinde@windriver.com>

This patch adds similar optimizations to the C++ FE as have been
implemented earlier in the C FE.
The libcpp hunk enables use of CPP_EMBED token even for C++, not just
C; the preprocessor guarantees there is always a CPP_NUMBER CPP_COMMA
before CPP_EMBED and CPP_COMMA CPP_NUMBER after it which simplifies
parsing (unless #embed is more than 2GB, in that case it could be
CPP_NUMBER CPP_COMMA CPP_EMBED CPP_COMMA CPP_EMBED CPP_COMMA CPP_EMBED
CPP_COMMA CPP_NUMBER etc. with each CPP_EMBED covering at most INT_MAX
bytes).
Similarly to the C patch, this patch parses it into RAW_DATA_CST tree
in the braced initializers (and from there peels into INTEGER_CSTs unless
it is an initializer of an std::byte array or integral array with CHAR_BIT
element precision), parses CPP_EMBED in cp_parser_expression into just
the last INTEGER_CST in it because I think users don't need millions of
-Wunused-value warnings because they did useless
  int a = (
  #embed "megabyte.dat"
  );
and so most of the inner INTEGER_CSTs would be there just for the warning,
and in the rest of contexts like template argument list, function argument
list, attribute argument list, ...) parse it into a sequence of INTEGER_CSTs
(I wrote a range/iterator classes to simplify that).

My dumb
cat embed-11.c
constexpr unsigned char a[] = {
  #embed "cc1plus"
};
const unsigned char *b = a;
testcase where cc1plus is 492329008 bytes long when configured
--enable-checking=yes,rtl,extra against recent binutils with .base64 gas
support results in:
time ./xg++ -B ./ -S -O2 embed-11.c

real    0m4.350s
user    0m2.427s
sys     0m0.830s
time ./xg++ -B ./ -c -O2 embed-11.c

real    0m6.932s
user    0m6.034s
sys     0m0.888s
(compared to running out of memory or very long compilation).
On a shorter inclusion,
cat embed-12.c
constexpr unsigned char a[] = {
  #embed "xg++"
};
const unsigned char *b = a;
where xg++ is 15225904 bytes long, this takes using GCC with the #embed
patchset except for this patch:
time ~/src/gcc/obj36/gcc/xg++ -B ~/src/gcc/obj36/gcc/ -S -O2 embed-12.c

real    0m33.190s
user    0m32.327s
sys     0m0.790s
and with this patch:
time ./xg++ -B ./ -S -O2 embed-12.c

real    0m0.118s
user    0m0.090s
sys     0m0.028s

The patch doesn't change anything on what the first patch in the series
introduces even for C++, namely that #embed is expanded (actually or as if)
into a sequence of literals like
127,69,76,70,2,1,1,3,0,0,0,0,0,0,0,0,2,0,62,0,1,0,0,0,80,211,64,0,0,0,0,0,64,0,0,0,0,0,0,0,8,253
and so each element has int type.
That is how I believe it is in C23, and the different versions of the
C++ P1967 paper specified there some casts, P1967R12 in particular
"Otherwise, the integral constant expression is the value of std::fgetc’s return is cast
to unsigned char."
but please see
https://github.com/llvm/llvm-project/pull/97274#issuecomment-2230929277
comment and whether we really want the preprocessor to preprocess it for
C++ as (or as-if)
static_cast<unsigned char>(127),static_cast<unsigned char>(69),static_cast<unsigned char>(76),static_cast<unsigned char>(70),static_cast<unsigned char>(2),...
i.e. 9 tokens per byte rather than 2, or
(unsigned char)127,(unsigned char)69,...
or
((unsigned char)127),((unsigned char)69),...
etc.
Without a literal suffix for unsigned char constant literals it is horrible,
plus the incompatibility between C and C++.  Sure, we could use the magic
form more often for C++ to save the size and do the 9 or how many tokens
form only for the boundary constants and use #embed "." __gnu__::__base64__("...")
for what is in between if there are at least 2 tokens inside of it.
E.g. (unsigned char)127 vs. static_cast<unsigned char>(127) behaves
differently if there is constexpr long long p[] = { ... };
...
  #embed __FILE__
[p]

2024-12-06  Jakub Jelinek  <jakub@redhat.com>

libcpp/
	* files.cc (finish_embed): Use CPP_EMBED even for C++.
gcc/
	* tree.h (RAW_DATA_UCHAR_ELT, RAW_DATA_SCHAR_ELT): Define.
gcc/cp/ChangeLog:
	* cp-tree.h (class raw_data_iterator): New type.
	(class raw_data_range): New type.
	* parser.cc (cp_parser_postfix_open_square_expression): Handle
	parsing of CPP_EMBED.
	(cp_parser_parenthesized_expression_list): Likewise.  Use
	cp_lexer_next_token_is.
	(cp_parser_expression): Handle parsing of CPP_EMBED.
	(cp_parser_template_argument_list): Likewise.
	(cp_parser_initializer_list): Likewise.
	(cp_parser_oacc_clause_tile): Likewise.
	(cp_parser_omp_tile_sizes): Likewise.
	* pt.cc (tsubst_expr): Handle RAW_DATA_CST.
	* constexpr.cc (reduced_constant_expression_p): Likewise.
	(raw_data_cst_elt): New function.
	(find_array_ctor_elt): Handle RAW_DATA_CST.
	(cxx_eval_array_reference): Likewise.
	* typeck2.cc (digest_init_r): Emit -Wnarrowing and/or -Wconversion
	diagnostics.
	(process_init_constructor_array): Handle RAW_DATA_CST.
	* decl.cc (maybe_deduce_size_from_array_init): Likewise.
	(is_direct_enum_init): Fail for RAW_DATA_CST.
	(cp_maybe_split_raw_data): New function.
	(consume_init): New function.
	(reshape_init_array_1): Add VECTOR_P argument.  Handle RAW_DATA_CST.
	(reshape_init_array): Adjust reshape_init_array_1 caller.
	(reshape_init_vector): Likewise.
	(reshape_init_class): Handle RAW_DATA_CST.
	(reshape_init_r): Likewise.
gcc/testsuite/
	* c-c++-common/cpp/embed-22.c: New test.
	* c-c++-common/cpp/embed-23.c: New test.
	* g++.dg/cpp/embed-4.C: New test.
	* g++.dg/cpp/embed-5.C: New test.
	* g++.dg/cpp/embed-6.C: New test.
	* g++.dg/cpp/embed-7.C: New test.
	* g++.dg/cpp/embed-8.C: New test.
	* g++.dg/cpp/embed-9.C: New test.
	* g++.dg/cpp/embed-10.C: New test.
	* g++.dg/cpp/embed-11.C: New test.
	* g++.dg/cpp/embed-12.C: New test.
	* g++.dg/cpp/embed-13.C: New test.
	* g++.dg/cpp/embed-14.C: New test.

I enabled include translation to header units in r15-1104-ga29f481bbcaf2b,
but it seems that patch wasn't sufficient, as any diagnostics in the main
source file would show up as coming from the header instead.

Fixed by setting buffer->file for leaving the file transition that my
previous patch made us enter.  And don't push a buffer of newlines, in this
case that messes up line numbers instead of aligning them.

libcpp/ChangeLog:

	* files.cc (_cpp_stack_file): Handle -include of header unit more
	specially.

gcc/testsuite/ChangeLog:

	* g++.dg/modules/dashinclude-1_b.C: Add an #error.
	* g++.dg/modules/dashinclude-1_a.H: Remove dg-module-do run.

The dependency output for header unit modules is based on the absolute
pathname of the header file, but that's not something that a makefile can
portably refer to.  This patch adds a .c++-header-unit target based on the
header name relative to an element of the include path.

libcpp/ChangeLog:

	* internal.h (_cpp_get_file_dir): Declare.
	* files.cc (_cpp_get_file_dir): New fn.
	* mkdeps.cc (make_write): Use it.

gcc/testsuite/ChangeLog:

	* g++.dg/modules/dep-4.H: New test.

The C++ modules code has a -fmodule-header (or -x c++-{user,system}-header)
option to specify looking up headers to compile to header units on the usual
include paths.  I'd like to have the same functionality for full C++20
modules such as module std, which I proposed to live on the include path at
bits/std.cc.  But this behavior doesn't seem necessarily connected to
modules, so I'm proposing a general C/C++ option to specify the behavior of
looking in the include path for the input files specified on the command
line.

Other ideas for the name of the option are very welcome.

The libcpp change is to allow -fsearch-include-path{,=user} to find files in
the current working directory, like -include.  This can be handy for a quick
compile of both std.cc and a file that imports it, e.g.

g++ -std=c++20 -fmodules -fsearch-include-path bits/std.cc importer.cc

gcc/ChangeLog:

	* doc/cppopts.texi: Document -fsearch-include-path.
	* doc/invoke.texi: Mention it for modules.

gcc/c-family/ChangeLog:

	* c.opt: Add -fsearch-include-path.
	* c-opts.cc (c_common_post_options): Handle it.

gcc/cp/ChangeLog:

	* module.cc (module_preprocess_options): Don't override it.

libcpp/ChangeLog:

	* internal.h (search_path_head): Declare.
	* files.cc (search_path_head): No longer static.
	* init.cc (cpp_read_main_file): Use it.

I've tried to build stage3 with
-Wleading-whitespace=blanks -Wtrailing-whitespace=blank -Wno-error=leading-whitespace=blanks -Wno-error=trailing-whitespace=blank
added to STRICT_WARN and that expectably resulted in about
2744 unique trailing whitespace warnings and 124837 leading whitespace
warnings when excluding *.md files (which obviously is in big part a
generator issue).  Others from that are generator related, I think those
need to be solved later.

The following patch just fixes up the easy case (trailing whitespace),
which could be easily automated:
for i in `find . -name \*.h -o -name \*.cc -o -name \*.c | xargs grep -l '[ 	]$' | grep -v testsuite/`; do sed -i -e 's/[ 	]*$//' $i; done
I've excluded files which I knew are obviously generated or go FE.

Is there anything else we'd want to avoid the changes?

Due to patch size, I've split it between gcc/ part
and rest (include/, libiberty/, libgcc/, libcpp/, libstdc++-v3/;
this part).

2024-10-24  Jakub Jelinek  <jakub@redhat.com>

include/
	* dyn-string.h: Remove trailing whitespace.
	* libiberty.h: Likewise.
	* xregex.h: Likewise.
	* splay-tree.h: Likewise.
	* partition.h: Likewise.
	* plugin-api.h: Likewise.
	* demangle.h: Likewise.
	* vtv-change-permission.h: Likewise.
	* fibheap.h: Likewise.
	* hsa_ext_image.h: Likewise.
	* hashtab.h: Likewise.
	* libcollector.h: Likewise.
	* sort.h: Likewise.
	* symcat.h: Likewise.
	* hsa_ext_amd.h: Likewise.
libcpp/
	* directives.cc: Remove trailing whitespace.
	* mkdeps.cc: Likewise.
	* line-map.cc: Likewise.
	* internal.h: Likewise.
	* files.cc: Likewise.
	* init.cc: Likewise.
	* makeucnid.cc: Likewise.
	* system.h: Likewise.
	* include/line-map.h: Likewise.
	* include/symtab.h: Likewise.
	* include/cpplib.h: Likewise.
	* expr.cc: Likewise.
	* charset.cc: Likewise.
	* macro.cc: Likewise.
	* errors.cc: Likewise.
	* lex.cc: Likewise.
	* traditional.cc: Likewise.
libgcc/
	* crtstuff.c: Remove trailing whitespace.
	* libgcov.h: Likewise.
	* config/alpha/crtfastmath.c: Likewise.
	* config/alpha/vms-gcc_shell_handler.c: Likewise.
	* config/alpha/vms-unwind.h: Likewise.
	* config/pa/linux-atomic.c: Likewise.
	* config/pa/linux-unwind.h: Likewise.
	* config/pa/quadlib.c: Likewise.
	* config/pa/fptr.c: Likewise.
	* config/s390/32/_fixsfdi.c: Likewise.
	* config/s390/32/_fixunssfdi.c: Likewise.
	* config/s390/32/_fixunsdfdi.c: Likewise.
	* config/c6x/pr-support.c: Likewise.
	* config/lm32/_udivsi3.c: Likewise.
	* config/lm32/libgcc_lm32.h: Likewise.
	* config/lm32/_udivmodsi4.c: Likewise.
	* config/lm32/_mulsi3.c: Likewise.
	* config/lm32/_modsi3.c: Likewise.
	* config/lm32/_umodsi3.c: Likewise.
	* config/lm32/_divsi3.c: Likewise.
	* config/darwin-crt3.c: Likewise.
	* config/msp430/mpy.c: Likewise.
	* config/ia64/tf-signs.c: Likewise.
	* config/ia64/fde-vms.c: Likewise.
	* config/ia64/unwind-ia64.c: Likewise.
	* config/ia64/vms-unwind.h: Likewise.
	* config/ia64/sfp-exceptions.c: Likewise.
	* config/ia64/quadlib.c: Likewise.
	* config/ia64/unwind-ia64.h: Likewise.
	* config/rl78/vregs.h: Likewise.
	* config/arm/bpabi.c: Likewise.
	* config/arm/unwind-arm.c: Likewise.
	* config/arm/pr-support.c: Likewise.
	* config/arm/linux-atomic.c: Likewise.
	* config/arm/bpabi-lib.h: Likewise.
	* config/frv/frvend.c: Likewise.
	* config/frv/cmovw.c: Likewise.
	* config/frv/frvbegin.c: Likewise.
	* config/frv/cmovd.c: Likewise.
	* config/frv/cmovh.c: Likewise.
	* config/aarch64/cpuinfo.c: Likewise.
	* config/i386/crtfastmath.c: Likewise.
	* config/i386/cygming-crtend.c: Likewise.
	* config/i386/32/tf-signs.c: Likewise.
	* config/i386/crtprec.c: Likewise.
	* config/i386/sfp-exceptions.c: Likewise.
	* config/i386/w32-unwind.h: Likewise.
	* config/m32r/initfini.c: Likewise.
	* config/sparc/crtfastmath.c: Likewise.
	* config/gcn/amdgcn_veclib.h: Likewise.
	* config/nios2/linux-atomic.c: Likewise.
	* config/nios2/linux-unwind.h: Likewise.
	* config/nios2/lib2-mul.c: Likewise.
	* config/nios2/lib2-nios2.h: Likewise.
	* config/xtensa/unwind-dw2-xtensa.c: Likewise.
	* config/rs6000/darwin-fallback.c: Likewise.
	* config/rs6000/ibm-ldouble.c: Likewise.
	* config/rs6000/sfp-machine.h: Likewise.
	* config/rs6000/darwin-asm.h: Likewise.
	* config/rs6000/darwin-crt2.c: Likewise.
	* config/rs6000/aix-unwind.h: Likewise.
	* config/rs6000/sfp-exceptions.c: Likewise.
	* config/gthr-vxworks.c: Likewise.
	* config/riscv/atomic.c: Likewise.
	* config/visium/memcpy.c: Likewise.
	* config/darwin-crt-tm.c: Likewise.
	* config/stormy16/lib2funcs.c: Likewise.
	* config/arc/ieee-754/divtab-arc-sf.c: Likewise.
	* config/arc/ieee-754/divtab-arc-df.c: Likewise.
	* config/arc/initfini.c: Likewise.
	* config/sol2/gmon.c: Likewise.
	* config/microblaze/divsi3_table.c: Likewise.
	* config/m68k/fpgnulib.c: Likewise.
	* libgcov-driver.c: Likewise.
	* unwind-dw2.c: Likewise.
	* fp-bit.c: Likewise.
	* dfp-bit.h: Likewise.
	* dfp-bit.c: Likewise.
	* libgcov-driver-system.c: Likewise.
libgcc/config/libbid/
	* _le_td.c: Remove trailing whitespace.
	* bid128_compare.c: Likewise.
	* bid_div_macros.h: Likewise.
	* bid64_to_bid128.c: Likewise.
	* bid64_to_uint32.c: Likewise.
	* bid128_to_uint64.c: Likewise.
	* bid64_div.c: Likewise.
	* bid128_round_integral.c: Likewise.
	* bid_binarydecimal.c: Likewise.
	* bid128_string.c: Likewise.
	* bid_flag_operations.c: Likewise.
	* bid128_to_int64.c: Likewise.
	* _mul_sd.c: Likewise.
	* bid64_mul.c: Likewise.
	* bid128_noncomp.c: Likewise.
	* _gt_dd.c: Likewise.
	* bid64_add.c: Likewise.
	* bid64_string.c: Likewise.
	* bid_from_int.c: Likewise.
	* bid128.c: Likewise.
	* _ge_dd.c: Likewise.
	* _ne_sd.c: Likewise.
	* _dd_to_td.c: Likewise.
	* _unord_sd.c: Likewise.
	* bid64_to_uint64.c: Likewise.
	* _gt_sd.c: Likewise.
	* _sd_to_td.c: Likewise.
	* _addsub_td.c: Likewise.
	* _ne_td.c: Likewise.
	* bid_dpd.c: Likewise.
	* bid128_add.c: Likewise.
	* bid128_next.c: Likewise.
	* _lt_sd.c: Likewise.
	* bid64_next.c: Likewise.
	* bid128_mul.c: Likewise.
	* _lt_dd.c: Likewise.
	* _ge_td.c: Likewise.
	* _unord_dd.c: Likewise.
	* bid64_sqrt.c: Likewise.
	* bid_sqrt_macros.h: Likewise.
	* bid64_fma.c: Likewise.
	* _sd_to_dd.c: Likewise.
	* bid_conf.h: Likewise.
	* bid64_noncomp.c: Likewise.
	* bid_gcc_intrinsics.h: Likewise.
	* _gt_td.c: Likewise.
	* _ge_sd.c: Likewise.
	* bid128_minmax.c: Likewise.
	* bid128_quantize.c: Likewise.
	* bid32_to_bid64.c: Likewise.
	* bid_round.c: Likewise.
	* _td_to_sd.c: Likewise.
	* bid_inline_add.h: Likewise.
	* bid128_fma.c: Likewise.
	* _eq_td.c: Likewise.
	* bid32_to_bid128.c: Likewise.
	* bid64_rem.c: Likewise.
	* bid128_2_str_tables.c: Likewise.
	* _mul_dd.c: Likewise.
	* _dd_to_sd.c: Likewise.
	* bid128_div.c: Likewise.
	* _lt_td.c: Likewise.
	* bid64_compare.c: Likewise.
	* bid64_to_int32.c: Likewise.
	* _unord_td.c: Likewise.
	* bid128_rem.c: Likewise.
	* bid_internal.h: Likewise.
	* bid64_to_int64.c: Likewise.
	* _eq_dd.c: Likewise.
	* _td_to_dd.c: Likewise.
	* bid128_to_int32.c: Likewise.
	* bid128_to_uint32.c: Likewise.
	* _ne_dd.c: Likewise.
	* bid64_quantize.c: Likewise.
	* _le_dd.c: Likewise.
	* bid64_round_integral.c: Likewise.
	* _le_sd.c: Likewise.
	* bid64_minmax.c: Likewise.
libgcc/config/avr/libf7/
	* f7-renames.h: Remove trailing whitespace.
libstdc++-v3/
	* include/debug/debug.h: Remove trailing whitespace.
	* include/parallel/base.h: Likewise.
	* include/parallel/types.h: Likewise.
	* include/parallel/settings.h: Likewise.
	* include/parallel/multiseq_selection.h: Likewise.
	* include/parallel/partition.h: Likewise.
	* include/parallel/random_number.h: Likewise.
	* include/parallel/find_selectors.h: Likewise.
	* include/parallel/partial_sum.h: Likewise.
	* include/parallel/list_partition.h: Likewise.
	* include/parallel/search.h: Likewise.
	* include/parallel/algorithmfwd.h: Likewise.
	* include/parallel/random_shuffle.h: Likewise.
	* include/parallel/multiway_mergesort.h: Likewise.
	* include/parallel/sort.h: Likewise.
	* include/parallel/algobase.h: Likewise.
	* include/parallel/numericfwd.h: Likewise.
	* include/parallel/multiway_merge.h: Likewise.
	* include/parallel/losertree.h: Likewise.
	* include/bits/basic_ios.h: Likewise.
	* include/bits/stringfwd.h: Likewise.
	* include/bits/ostream_insert.h: Likewise.
	* include/bits/stl_heap.h: Likewise.
	* include/bits/unordered_map.h: Likewise.
	* include/bits/hashtable_policy.h: Likewise.
	* include/bits/stl_iterator_base_funcs.h: Likewise.
	* include/bits/valarray_before.h: Likewise.
	* include/bits/regex.h: Likewise.
	* include/bits/postypes.h: Likewise.
	* include/bits/stl_iterator.h: Likewise.
	* include/bits/localefwd.h: Likewise.
	* include/bits/stl_algo.h: Likewise.
	* include/bits/ios_base.h: Likewise.
	* include/bits/stl_function.h: Likewise.
	* include/bits/basic_string.h: Likewise.
	* include/bits/hashtable.h: Likewise.
	* include/bits/valarray_after.h: Likewise.
	* include/bits/char_traits.h: Likewise.
	* include/bits/gslice.h: Likewise.
	* include/bits/locale_facets_nonio.h: Likewise.
	* include/bits/mask_array.h: Likewise.
	* include/bits/specfun.h: Likewise.
	* include/bits/random.h: Likewise.
	* include/bits/slice_array.h: Likewise.
	* include/bits/valarray_array.h: Likewise.
	* include/tr1/float.h: Likewise.
	* include/tr1/functional_hash.h: Likewise.
	* include/tr1/math.h: Likewise.
	* include/tr1/hashtable_policy.h: Likewise.
	* include/tr1/stdio.h: Likewise.
	* include/tr1/complex.h: Likewise.
	* include/tr1/stdbool.h: Likewise.
	* include/tr1/stdarg.h: Likewise.
	* include/tr1/inttypes.h: Likewise.
	* include/tr1/fenv.h: Likewise.
	* include/tr1/stdlib.h: Likewise.
	* include/tr1/wchar.h: Likewise.
	* include/tr1/tgmath.h: Likewise.
	* include/tr1/limits.h: Likewise.
	* include/tr1/wctype.h: Likewise.
	* include/tr1/stdint.h: Likewise.
	* include/tr1/ctype.h: Likewise.
	* include/tr1/random.h: Likewise.
	* include/tr1/shared_ptr.h: Likewise.
	* include/ext/mt_allocator.h: Likewise.
	* include/ext/sso_string_base.h: Likewise.
	* include/ext/debug_allocator.h: Likewise.
	* include/ext/vstring_fwd.h: Likewise.
	* include/ext/pointer.h: Likewise.
	* include/ext/pod_char_traits.h: Likewise.
	* include/ext/malloc_allocator.h: Likewise.
	* include/ext/vstring.h: Likewise.
	* include/ext/bitmap_allocator.h: Likewise.
	* include/ext/pool_allocator.h: Likewise.
	* include/ext/type_traits.h: Likewise.
	* include/ext/ropeimpl.h: Likewise.
	* include/ext/codecvt_specializations.h: Likewise.
	* include/ext/throw_allocator.h: Likewise.
	* include/ext/extptr_allocator.h: Likewise.
	* include/ext/atomicity.h: Likewise.
	* include/ext/concurrence.h: Likewise.
	* include/c_compatibility/wchar.h: Likewise.
	* include/c_compatibility/stdint.h: Likewise.
	* include/backward/hash_fun.h: Likewise.
	* include/backward/binders.h: Likewise.
	* include/backward/hashtable.h: Likewise.
	* include/backward/auto_ptr.h: Likewise.
	* libsupc++/eh_arm.cc: Likewise.
	* libsupc++/unwind-cxx.h: Likewise.
	* libsupc++/si_class_type_info.cc: Likewise.
	* libsupc++/vec.cc: Likewise.
	* libsupc++/class_type_info.cc: Likewise.
	* libsupc++/vmi_class_type_info.cc: Likewise.
	* libsupc++/guard_error.cc: Likewise.
	* libsupc++/bad_typeid.cc: Likewise.
	* libsupc++/eh_personality.cc: Likewise.
	* libsupc++/atexit_arm.cc: Likewise.
	* libsupc++/pmem_type_info.cc: Likewise.
	* libsupc++/vterminate.cc: Likewise.
	* libsupc++/eh_terminate.cc: Likewise.
	* libsupc++/bad_cast.cc: Likewise.
	* libsupc++/exception_ptr.h: Likewise.
	* libsupc++/eh_throw.cc: Likewise.
	* libsupc++/bad_alloc.cc: Likewise.
	* libsupc++/nested_exception.cc: Likewise.
	* libsupc++/pointer_type_info.cc: Likewise.
	* libsupc++/pbase_type_info.cc: Likewise.
	* libsupc++/bad_array_new.cc: Likewise.
	* libsupc++/pure.cc: Likewise.
	* libsupc++/eh_exception.cc: Likewise.
	* libsupc++/bad_array_length.cc: Likewise.
	* libsupc++/cxxabi.h: Likewise.
	* libsupc++/guard.cc: Likewise.
	* libsupc++/eh_catch.cc: Likewise.
	* libsupc++/cxxabi_forced.h: Likewise.
	* libsupc++/tinfo.h: Likewise.

This patch actually optimizes #embed, so far in C.

For a simple testcase (for 494447200 bytes long cc1plus):
cat embed-11.c
unsigned char a[] = {
  #embed "cc1plus"
};
time ./xgcc -B ./ -S -std=c23 -O2 embed-11.c

real    0m13.647s
user    0m7.157s
sys     0m2.597s
time ./xgcc -B ./ -c -std=c23 -O2 embed-11.c

real    0m28.649s
user    0m26.653s
sys     0m1.958s

and when configured against binutils with .base64 support
time ./xgcc -B ./ -S -std=c23 -O2 embed-11.c

real    0m4.283s
user    0m2.288s
sys     0m0.859s
time ./xgcc -B ./ -c -std=c23 -O2 embed-11.c

real    0m6.888s
user    0m5.876s
sys     0m1.002s

(all times with --enable-checking=yes,rtl,extra compiler).

Even just
./cc1plus -E -o embed-11.i embed-11.c
(which doesn't have this optimization yet and so preprocesses it as
1.3GB preprocessed file) needed almost 25GB of compile time RAM (but
preprocessed fine).
And compiling that embed-11.i with -std=c23 -O0 by unpatched gcc
I gave up after 400 seconds when it already ate 45GB of RAM and didn't
produce a single byte into embed-11.s yet.

The patch introduces a new CPP_EMBED token which contains raw memory image
virtually representing a sequence of int literals.
To simplify the parsing complexities, the preprocessor guarantees CPP_EMBED
is only emitted if there are 4+ (it actually does that for 64+ right now)
literals in the sequence and emits CPP_NUMBER CPP_COMMA CPP_EMBED CPP_COMMA
CPP_NUMBER tokens (with more CPP_EMBED separated by CPP_COMMA if it is
longer than 2GB, as STRING_CSTs in GCC and also the new RAW_DATA_CST etc.
are limited to INT_MAX elements).  The main reason is that the preprocessor
doesn't really know in which context #embed directive appears, there could
be e.g.
{ 25 *
  #embed "whatever"
* 2 - 15 }
or similar and dealing with this special case deep in the expression parsing
is undesirable.
With the CPP_NUMBERs around it, I believe in the C FE the only places which
need handling of the CPP_EMBED token are initializer parsing (that is the
only one which adds actual optimizations for it), comma expressions (I
believe nothing really cares whether it is 25,13,95 or
25,13,0,1,2,3,4,5,6,7,8,9,10,13,95 etc., so besides the 2 outer CPP_NUMBER
the parsing just adds one INTEGER_CST to the comma expression, I doubt users
want to be spammed with millions of -Wunused warnings per #embed),
whatever uses c_parser_expr_list (function calls, attribute arguments,
OpenMP sizes clause argument, OpenACC tile clause argument and whatever uses
c_parser_get_builtin_args (mainly for __builtin_shufflevector).  Please correct
me if I'm wrong.

The patch introduces a RAW_DATA_CST tree code, which can then be used inside
of array CONSTRUCTOR elt values.  In some sense RAW_DATA_CST is similar to
STRING_CST, but right now STRING_CST is used only if the whole array
initializer is that constant, while RAW_DATA_CST at index idx (should be
always INTEGER_CST index, another advantage of the CPP_NUMBER around is that
[30 ... 250] =
  #embed "whatever"
really does what it would do with a integer sequence there) stands for
[idx] = RAW_DATA_POINTER (val)[0],
[idx+1] = RAW_DATA_POINTER (val)[1],
...
[idx+RAW_DATA_LENGTH (val)-1] = RAW_DATA_POINTER (val)[RAW_DATA_LENGTH (val)-1].
Another important thing is that unlike STRING_CST which has the data
embedded in it RAW_DATA_CST doesn't own the data, it has RAW_DATA_OWNER
which owns the data (that can be a STRING_CST, e.g. used for PCH or LTO
after reading LTO in) or another RAW_DATA_CST (with NULL RAW_DATA_OWNER,
standing for data owned by libcpp buffers).  The advantage is that it can be
cheaply peeled off, or split into multiple smaller pieces, e.g. if one uses
designated initializer to store something into the middle of a 10GB #embed
array, in no case we need to actually copy data around for that.
Right now RAW_DATA_CST is only used in initializers of integral arrays where
the integer type has (host) CHAR_BIT precision, so usually char/signed
char/unsigned char (for C++ later maybe std::byte); in theory we could say
allocate 4 times as big buffer for conversions to int array and depending
on endianity and storage order reversal etc., but I'm not sure if that is
something that will be actually needed in the wild.
And an optimization inside of c-common.cc attempts to undo that CPP_NUMBER
CPP_EMBED CPP_NUMBER division in case one uses #embed the usual way and
doesn't use the boundary literals in weird ways and the values there match
the surrounding bytes in the owner buffer.

For LTO, in order to avoid copying perhaps gigabytes long data around,
the hacks in the streamer out/in cause the data owned by libcpp to be
streamed right into the stream and streamed back as a STRING_CST which
owns the data.

2024-10-16  Jakub Jelinek  <jakub@redhat.com>

libcpp/
	* include/cpplib.h (TTYPE_TABLE): Add CPP_EMBED token type.
	* files.cc (finish_embed): For limit >= 64 and C preprocessing
	instead of emitting CPP_NUMBER CPP_COMMA separated sequence for the
	whole embed emit it just for the first and last byte and in between
	emit a CPP_EMBED token or tokens if too large.
gcc/
	* treestruct.def (TS_RAW_DATA_CST): New.
	* tree.def (RAW_DATA_CST): New tree code.
	* tree-core.h (struct tree_raw_data): New type.
	(union tree_node): Add raw_data_cst member.
	* tree.h (RAW_DATA_LENGTH, RAW_DATA_POINTER, RAW_DATA_OWNER): Define.
	(gt_ggc_mx, gt_pch_nx): Declare overloads for tree_raw_data *.
	* tree.cc (tree_node_structure_for_code): Handle RAW_DATA_CST.
	(initialize_tree_contains_struct): Handle TS_RAW_DATA_CST.
	(tree_code_size): Handle RAW_DATA_CST.
	(initializer_zerop): Likewise.
	(gt_ggc_mx, gt_pch_nx): Define overloads for tree_raw_data *.
	* gimplify.cc (gimplify_init_ctor_eval): Handle RAW_DATA_CST.
	* fold-const.cc (operand_compare::operand_equal_p): Handle
	RAW_DATA_CST.  Formatting fix.
	(operand_compare::hash_operand): Handle RAW_DATA_CST.
	(native_encode_initializer): Likewise.
	(get_array_ctor_element_at_index): Likewise.
	(fold): Likewise.
	* gimple-fold.cc (fold_array_ctor_reference): Likewise.  Formatting
	fix.
	* varasm.cc (const_hash_1): Handle RAW_DATA_CST.
	(initializer_constant_valid_p_1): Likewise.
	(array_size_for_constructor): Likewise.
	(output_constructor_regular_field): Likewise.
	* expr.cc (categorize_ctor_elements_1): Likewise.
	(expand_expr_real_1) <case ARRAY_REF>: Punt for RAW_DATA_CST.
	* tree-streamer.cc (streamer_check_handled_ts_structures): Mark
	TS_RAW_DATA_CST as handled.
	* tree-streamer-in.cc (streamer_alloc_tree): Handle RAW_DATA_CST.
	(lto_input_ts_raw_data_cst_tree_pointers): New function.
	(streamer_read_tree_body): Call it for RAW_DATA_CST.
	* tree-streamer-out.cc (write_ts_raw_data_cst_tree_pointers): New
	function.
	(streamer_write_tree_body): Call it for RAW_DATA_CST.
	(streamer_write_tree_header): Handle RAW_DATA_CST.
	* lto-streamer-out.cc (DFS::DFS_write_tree_body): Handle RAW_DATA_CST.
	* tree-pretty-print.cc (dump_generic_node): Likewise.
gcc/c-family/
	* c-ppoutput.cc (token_streamer::stream): Add special code to spell
	CPP_EMBED token.
	* c-lex.cc (c_lex_with_flags): Handle CPP_EMBED.  Formatting fix.
	* c-common.cc (c_parse_error): Handle CPP_EMBED.
	(braced_list_to_string): Optimize RAW_DATA_CST surrounded by
	INTEGER_CSTs which match some bytes before or after RAW_DATA_CST in
	its owner.
gcc/c/
	* c-parser.cc (c_parser_braced_init): Handle CPP_EMBED.
	(c_parser_get_builtin_args): Likewise.
	(c_parser_expression): Likewise.
	(c_parser_expr_list): Likewise.
	* c-typeck.cc (digest_init): Handle RAW_DATA_CST.  Formatting fix.
	(init_node_successor): New function.
	(add_pending_init): Handle RAW_DATA_CST.
	(set_nonincremental_init): Formatting fix.
	(output_init_element): Handle RAW_DATA_CST.  Formatting fixes.
	(maybe_split_raw_data): New function.
	(process_init_element): Use maybe_split_raw_data.  Handle
	RAW_DATA_CST.
gcc/testsuite/
	* c-c++-common/cpp/embed-20.c: New test.
	* c-c++-common/cpp/embed-21.c: New test.
	* c-c++-common/cpp/embed-28.c: New test.
	* gcc.dg/cpp/embed-8.c: New test.
	* gcc.dg/cpp/embed-9.c: New test.
	* gcc.dg/cpp/embed-10.c: New test.
	* gcc.dg/cpp/embed-11.c: New test.
	* gcc.dg/cpp/embed-12.c: New test.
	* gcc.dg/cpp/embed-13.c: New test.
	* gcc.dg/cpp/embed-14.c: New test.
	* gcc.dg/cpp/embed-15.c: New test.
	* gcc.dg/cpp/embed-16.c: New test.
	* gcc.dg/pch/embed-1.c: New test.
	* gcc.dg/pch/embed-1.hs: New test.
	* gcc.dg/lto/embed-1_0.c: New test.
	* gcc.dg/lto/embed-1_1.c: New test.

When working on #embed support, or -Wheader-guard or other recent libcpp
changes, I've been annoyed by the libcpp diagnostics being visually
different from normal gcc diagnostics, especially in the area of quoting
stuff in the diagnostic messages.
Normall GCC diagnostics is gcc_diag/gcc_tdiag, one can use
%</%>, %qs etc. in there, while libcpp diagnostics was marked as printf
and in libcpp we've been very creative with quoting stuff, either
no quotes at all, or "something" quoting, or 'something' quoting, or
`something' quoting (but in none of the cases it used colors consistently
with the rest of the compiler).

Now, libcpp diagnostics is always emitted using a callback,
pfile->cb.diagnostic.  On the gcc/ side, this callback is initialized with
genmatch.cc:  cb->diagnostic = diagnostic_cb;
c-family/c-opts.cc:  cb->diagnostic = c_cpp_diagnostic;
fortran/cpp.cc:  cb->diagnostic = cb_cpp_diagnostic;
where the latter two just use diagnostic_report_diagnostic, so actually
support all the gcc_diag stuff, only the genmatch.cc case didn't.

So, the following patch changes genmatch.cc to use pp_format* instead
of vfprintf so that it supports the gcc_diag formatting (pretty-print.o
unfortunately has various dependencies, so had to link genmatch with
libcommon.a libbacktrace.a and tweak Makefile.in so that there are no
circular dependencies) and marks the libcpp diagnostic routines as
gcc_diag rather than printf.  That change resulted in hundreds of
-Wformat-diag new warnings (most of them useful and resulting IMHO in
better diagnostics), so the rest of the patch is changing the format
strings to make -Wformat-diag happy and adjusting the testsuite for
the differences in how is the diagnostic reformatted.

Dunno if some out of GCC tree projects use libcpp, that case would
make it harder because one couldn't use vfprintf in the diagnostic
callback anymore, but there is always David's libdiagnostic which could
be used for that purpose IMHO.

2024-10-12  Jakub Jelinek  <jakub@redhat.com>

libcpp/
	* include/cpplib.h (ATTRIBUTE_CPP_PPDIAG): Define.
	(struct cpp_callbacks): Use ATTRIBUTE_CPP_PPDIAG instead of
	ATTRIBUTE_FPTR_PRINTF on diagnostic callback.
	(cpp_error, cpp_warning, cpp_pedwarning, cpp_warning_syshdr): Use
	ATTRIBUTE_CPP_PPDIAG (3, 4) instead of ATTRIBUTE_PRINTF_3.
	(cpp_warning_at, cpp_pedwarning_at): Use ATTRIBUTE_CPP_PPDIAG (4, 5)
	instead of ATTRIBUTE_PRINTF_4.
	(cpp_error_with_line, cpp_warning_with_line, cpp_pedwarning_with_line,
	cpp_warning_with_line_syshdr): Use ATTRIBUTE_CPP_PPDIAG (5, 6)
	instead of ATTRIBUTE_PRINTF_5.
	(cpp_error_at): Use ATTRIBUTE_CPP_PPDIAG (4, 5) instead of
	ATTRIBUTE_PRINTF_4.
	* Makefile.in (po/$(PACKAGE).pot): Use --language=GCC-source rather
	than --language=c.
	* errors.cc (cpp_diagnostic_at, cpp_diagnostic,
	cpp_diagnostic_with_line): Use ATTRIBUTE_CPP_PPDIAG instead of
	-ATTRIBUTE_FPTR_PRINTF.
	* charset.cc (cpp_host_to_exec_charset, _cpp_valid_ucn, convert_hex,
	convert_oct, convert_escape): Fix up -Wformat-diag warnings.
	(cpp_interpret_string_ranges, count_source_chars): Use
	ATTRIBUTE_CPP_PPDIAG instead of ATTRIBUTE_FPTR_PRINTF.
	(narrow_str_to_charconst): Fix up -Wformat-diag warnings.
	* directives.cc (check_eol_1, directive_diagnostics, lex_macro_node,
	do_undef, glue_header_name, parse_include, do_include_common,
	do_include_next, _cpp_parse_embed_params, do_embed, read_flag,
	do_line, do_linemarker, register_pragma_1, do_pragma_once,
	do_pragma_push_macro, do_pragma_pop_macro, do_pragma_poison,
	do_pragma_system_header, do_pragma_warning_or_error, _cpp_do__Pragma,
	do_else, do_elif, do_endif, parse_answer, do_assert,
	cpp_define_unused): Likewise.
	* expr.cc (cpp_classify_number, parse_defined, eval_token,
	_cpp_parse_expr, reduce, check_promotion): Likewise.
	* files.cc (_cpp_find_file, finish_base64_embed,
	_cpp_pop_file_buffer): Likewise.
	* init.cc (sanity_checks): Likewise.
	* lex.cc (_cpp_process_line_notes, maybe_warn_bidi_on_char,
	_cpp_warn_invalid_utf8, _cpp_skip_block_comment,
	warn_about_normalization, forms_identifier_p, maybe_va_opt_error,
	identifier_diagnostics_on_lex, cpp_maybe_module_directive): Likewise.
	* macro.cc (class vaopt_state, builtin_has_include_1,
	builtin_has_include, builtin_has_embed, _cpp_warn_if_unused_macro,
	_cpp_builtin_macro_text, builtin_macro, stringify_arg,
	_cpp_arguments_ok, collect_args, enter_macro_context,
	_cpp_save_parameter, parse_params, create_iso_definition,
	_cpp_create_definition, check_trad_stringification): Likewise.
	* pch.cc (cpp_valid_state): Likewise.
	* traditional.cc (_cpp_scan_out_logical_line, recursive_macro):
	Likewise.
gcc/
	* Makefile.in (generated_files): Remove {gimple,generic}-match*.
	(generated_match_files): New variable.  Add a dependency of
	$(filter-out $(OBJS-libcommon),$(ALL_HOST_OBJS)) files on those.
	(build/genmatch$(build_exeext)): Depend on and link against
	libcommon.a and $(LIBBACKTRACE).
	* genmatch.cc: Include pretty-print.h and input.h.
	(ggc_internal_cleared_alloc, ggc_free): Remove.
	(fatal): New function.
	(line_table): Remove.
	(linemap_client_expand_location_to_spelling_point): Remove.
	(diagnostic_cb): Use gcc_diag rather than printf format.  Use
	pp_format_verbatim on a temporary pretty_printer instead of
	vfprintf.
	(fatal_at, warning_at): Use gcc_diag rather than printf format.
	(output_line_directive): Rename location_hash to loc_hash.
	(parser::eat_ident, parser::parse_operation, parser::parse_expr,
	parser::parse_pattern, parser::finish_match_operand): Fix up
	-Wformat-diag warnings.
gcc/c-family/
	* c-lex.cc (c_common_has_attribute,
	c_common_lex_availability_macro): Fix up -Wformat-diag warnings.
gcc/testsuite/
	* c-c++-common/cpp/counter-2.c: Adjust expected diagnostics for
	libcpp diagnostic formatting changes.
	* c-c++-common/cpp/embed-3.c: Likewise.
	* c-c++-common/cpp/embed-4.c: Likewise.
	* c-c++-common/cpp/embed-16.c: Likewise.
	* c-c++-common/cpp/embed-18.c: Likewise.
	* c-c++-common/cpp/eof-2.c: Likewise.
	* c-c++-common/cpp/eof-3.c: Likewise.
	* c-c++-common/cpp/fmax-include-depth.c: Likewise.
	* c-c++-common/cpp/has-builtin.c: Likewise.
	* c-c++-common/cpp/line-2.c: Likewise.
	* c-c++-common/cpp/line-3.c: Likewise.
	* c-c++-common/cpp/macro-arg-count-1.c: Likewise.
	* c-c++-common/cpp/macro-arg-count-2.c: Likewise.
	* c-c++-common/cpp/macro-ranges.c: Likewise.
	* c-c++-common/cpp/named-universal-char-escape-4.c: Likewise.
	* c-c++-common/cpp/named-universal-char-escape-5.c: Likewise.
	* c-c++-common/cpp/pr88974.c: Likewise.
	* c-c++-common/cpp/va-opt-error.c: Likewise.
	* c-c++-common/cpp/va-opt-pedantic.c: Likewise.
	* c-c++-common/cpp/Wheader-guard-2.c: Likewise.
	* c-c++-common/cpp/Wheader-guard-3.c: Likewise.
	* c-c++-common/cpp/Winvalid-utf8-1.c: Likewise.
	* c-c++-common/cpp/Winvalid-utf8-2.c: Likewise.
	* c-c++-common/cpp/Winvalid-utf8-3.c: Likewise.
	* c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-1.c:
	Likewise.
	* c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c:
	Likewise.
	* c-c++-common/pr68833-3.c: Likewise.
	* c-c++-common/raw-string-directive-1.c: Likewise.
	* gcc.dg/analyzer/named-constants-Wunused-macros.c: Likewise.
	* gcc.dg/binary-constants-4.c: Likewise.
	* gcc.dg/builtin-redefine.c: Likewise.
	* gcc.dg/cpp/19951025-1.c: Likewise.
	* gcc.dg/cpp/c11-warning-1.c: Likewise.
	* gcc.dg/cpp/c11-warning-2.c: Likewise.
	* gcc.dg/cpp/c11-warning-3.c: Likewise.
	* gcc.dg/cpp/c23-elifdef-2.c: Likewise.
	* gcc.dg/cpp/c23-warning-2.c: Likewise.
	* gcc.dg/cpp/embed-2.c: Likewise.
	* gcc.dg/cpp/embed-3.c: Likewise.
	* gcc.dg/cpp/embed-4.c: Likewise.
	* gcc.dg/cpp/expr.c: Likewise.
	* gcc.dg/cpp/gnu11-elifdef-2.c: Likewise.
	* gcc.dg/cpp/gnu11-elifdef-3.c: Likewise.
	* gcc.dg/cpp/gnu11-elifdef-4.c: Likewise.
	* gcc.dg/cpp/gnu11-warning-1.c: Likewise.
	* gcc.dg/cpp/gnu11-warning-2.c: Likewise.
	* gcc.dg/cpp/gnu11-warning-3.c: Likewise.
	* gcc.dg/cpp/gnu23-warning-2.c: Likewise.
	* gcc.dg/cpp/include6.c: Likewise.
	* gcc.dg/cpp/pr35322.c: Likewise.
	* gcc.dg/cpp/tr-warn6.c: Likewise.
	* gcc.dg/cpp/undef2.c: Likewise.
	* gcc.dg/cpp/warn-comments.c: Likewise.
	* gcc.dg/cpp/warn-comments-2.c: Likewise.
	* gcc.dg/cpp/warn-comments-3.c: Likewise.
	* gcc.dg/cpp/warn-cxx-compat.c: Likewise.
	* gcc.dg/cpp/warn-cxx-compat-2.c: Likewise.
	* gcc.dg/cpp/warn-deprecated.c: Likewise.
	* gcc.dg/cpp/warn-deprecated-2.c: Likewise.
	* gcc.dg/cpp/warn-long-long.c: Likewise.
	* gcc.dg/cpp/warn-long-long-2.c: Likewise.
	* gcc.dg/cpp/warn-normalized-1.c: Likewise.
	* gcc.dg/cpp/warn-normalized-2.c: Likewise.
	* gcc.dg/cpp/warn-normalized-3.c: Likewise.
	* gcc.dg/cpp/warn-normalized-4-bytes.c: Likewise.
	* gcc.dg/cpp/warn-normalized-4-unicode.c: Likewise.
	* gcc.dg/cpp/warn-redefined.c: Likewise.
	* gcc.dg/cpp/warn-redefined-2.c: Likewise.
	* gcc.dg/cpp/warn-traditional.c: Likewise.
	* gcc.dg/cpp/warn-traditional-2.c: Likewise.
	* gcc.dg/cpp/warn-trigraphs-1.c: Likewise.
	* gcc.dg/cpp/warn-trigraphs-2.c: Likewise.
	* gcc.dg/cpp/warn-trigraphs-3.c: Likewise.
	* gcc.dg/cpp/warn-trigraphs-4.c: Likewise.
	* gcc.dg/cpp/warn-undef.c: Likewise.
	* gcc.dg/cpp/warn-undef-2.c: Likewise.
	* gcc.dg/cpp/warn-unused-macros.c: Likewise.
	* gcc.dg/cpp/warn-unused-macros-2.c: Likewise.
	* gcc.dg/pch/counter-2.c: Likewise.
	* g++.dg/cpp0x/udlit-error1.C: Likewise.
	* g++.dg/cpp23/named-universal-char-escape1.C: Likewise.
	* g++.dg/cpp23/named-universal-char-escape2.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-1.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-2.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-3.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-4.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-5.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-6.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-7.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-8.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-9.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-10.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-11.C: Likewise.
	* g++.dg/cpp23/Winvalid-utf8-12.C: Likewise.
	* g++.dg/cpp/elifdef-3.C: Likewise.
	* g++.dg/cpp/elifdef-5.C: Likewise.
	* g++.dg/cpp/elifdef-6.C: Likewise.
	* g++.dg/cpp/elifdef-7.C: Likewise.
	* g++.dg/cpp/embed-1.C: Likewise.
	* g++.dg/cpp/embed-2.C: Likewise.
	* g++.dg/cpp/pedantic-errors.C: Likewise.
	* g++.dg/cpp/warning-1.C: Likewise.
	* g++.dg/cpp/warning-2.C: Likewise.
	* g++.dg/ext/bitint1.C: Likewise.
	* g++.dg/ext/bitint2.C: Likewise.