diff options
| author | michaelrj-google <71531609+michaelrj-google@users.noreply.github.com> | 2023-09-21 11:43:29 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-09-21 11:43:29 -0700 |
| commit | 5bd34e0a551e379d99004e5f34b932eb32569caa (patch) | |
| tree | 846d654f5669295db5d23f34826b63cad59d5467 /libc/utils/mathtools/ryu_tablegen.py | |
| parent | 9a99944df068b29b905cd8ba9a2132cc6382b6fb (diff) | |
[libc] Fix Off By One Errors In Printf Long Double (#66957)
Two major off-by-one errors are fixed in this patch. The first is in
float_to_string.h with length_for_num, which wasn't accounting for the
implicit leading bit when calculating the length of a number, causing
a missing digit on 80 bit float max. The other off-by-one is the
ryu_long_double_constants.h (a.k.a the Mega Table) not having any
entries for the last POW10_OFFSET in POW10_SPLIT. This was also found on
80 bit float max. Finally, the integer calculation mode was using a
slightly too short integer, again on 80 bit float max, not accounting
for the mantissa width. All of these are fixed in this patch.
Diffstat (limited to 'libc/utils/mathtools/ryu_tablegen.py')
| -rw-r--r-- | libc/utils/mathtools/ryu_tablegen.py | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/libc/utils/mathtools/ryu_tablegen.py b/libc/utils/mathtools/ryu_tablegen.py index e4b5734d95ce..91e2029e4177 100644 --- a/libc/utils/mathtools/ryu_tablegen.py +++ b/libc/utils/mathtools/ryu_tablegen.py @@ -173,10 +173,10 @@ if MOD_SIZE > (2**MID_INT_SIZE): ) else: print("static const uint64_t POW10_SPLIT[][" + str(MID_INT_SIZE // 64) + "] = {") - for idx in range(0, POSITIVE_ARR_SIZE): + for idx in range(0, POSITIVE_ARR_SIZE + 1): num_size = print_positive_table_for_idx(idx) + positive_size_arr[idx] = acc acc += num_size - positive_size_arr[idx + 1] = acc print("};") print( |