We currently raise a warning in `print_btf.py` when any of these types have a name. Linux kernel doesn't allow names in these types either.[0] However, there is nothing stopping frontends from giving names to these types. To make sure that they are always anonymous, explicitly skip the name emission. [0] https://elixir.bootlin.com/linux/v6.17.1/source/kernel/bpf/btf.c#L2586
Variant part, represented by `DW_TAG_variant_part` is a structure with a
discriminant and different variants, from which only one can be active
and valid at the same time. The discriminant is the main difference
between variant parts and unions represented by `DW_TAG_union` type.
Variant parts are used by Rust enums, which look like:
```rust
pub enum MyEnum {
First { a: u32, b: i32 },
Second(u32),
}
```
This type's debug info is the following `DICompositeType` with
`DW_TAG_structure_type` tag:
```llvm
!4 = !DICompositeType(tag: DW_TAG_structure_type, name: "MyEnum",
scope: !2, file: !5, size: 96, align: 32, flags: DIFlagPublic,
elements: !6, templateParams: !16,
identifier: "faba668fd9f71e9b7cf3b9ac5e8b93cb")
```
With one element being also a `DICompositeType`, but with
`DW_TAG_variant_part` tag:
```llvm
!6 = !{!7}
!7 = !DICompositeType(tag: DW_TAG_variant_part, scope: !4, file: !5,
size: 96, align: 32, elements: !8, templateParams: !16,
identifier: "e4aee046fc86d111657622fdcb8c42f7", discriminator: !21)
```
Which has a discriminator:
```llvm
!21 = !DIDerivedType(tag: DW_TAG_member, scope: !4, file: !5,
baseType: !13, size: 32, align: 32, flags: DIFlagArtificial)
```
Which then holds different variants as `DIDerivedType` elements with
`DW_TAG_member` tag:
```llvm
!8 = !{!9, !17}
!9 = !DIDerivedType(tag: DW_TAG_member, name: "First", scope: !7,
file: !5, baseType: !10, size: 96, align: 32, extraData: i32 0)
!10 = !DICompositeType(tag: DW_TAG_structure_type, name: "First",
scope: !4, file: !5, size: 96, align: 32, flags: DIFlagPublic,
elements: !11, templateParams: !16,
identifier: "cc7748c842e275452db4205b190c8ff7")
!11 = !{!12, !14}
!12 = !DIDerivedType(tag: DW_TAG_member, name: "a", scope: !10,
file: !5, baseType: !13, size: 32, align: 32, offset: 32,
flags: DIFlagPublic)
!13 = !DIBasicType(name: "u32", size: 32, encoding: DW_ATE_unsigned)
!14 = !DIDerivedType(tag: DW_TAG_member, name: "b", scope: !10,
file: !5, baseType: !15, size: 32, align: 32, offset: 64,
flags: DIFlagPublic)
!15 = !DIBasicType(name: "i32", size: 32, encoding: DW_ATE_signed)
!16 = !{}
!17 = !DIDerivedType(tag: DW_TAG_member, name: "Second", scope: !7,
file: !5, baseType: !18, size: 96, align: 32, extraData: i32 1)
!18 = !DICompositeType(tag: DW_TAG_structure_type, name: "Second",
scope: !4, file: !5, size: 96, align: 32, flags: DIFlagPublic,
elements: !19, templateParams: !16,
identifier: "a2094b1381f3082d504fbd0903aa7c06")
!19 = !{!20}
!20 = !DIDerivedType(tag: DW_TAG_member, name: "__0", scope: !18,
file: !5, baseType: !13, size: 32, align: 32, offset: 32,
flags: DIFlagPublic)
```
BPF backend was assuming that all the elements of any `DICompositeType`
have tag `DW_TAG_member` and are instances of `DIDerivedType`. However,
the single element of the outer composite type `!4` has tag
`DW_TAG_variant_part` and is an instance of `DICompositeType`. The
unconditional call of `cast<DIDerivedType>` on all elements was causing
an assertion failure when any Rust code with enums was compiled to the
BPF target.
Fix that by:
* Handling `DW_TAG_variant_part` in `visitStructType`.
* Replacing unconditional call of `cast<DIDerivedType>` over
`DICompositeType` elements with a `switch` statement, handling both
`DW_TAG_member` and `DW_TAG_variant_part` and casting the element to an
appropriate type (`DIDerivedType` or `DICompositeType`).
Fixes: https://github.com/llvm/llvm-project/issues/155778
getZExtValue() already returns uint64_t.
Fixes missing inner map struct type definitions [^1]. We should visit
the type of nested array of maps like we do for global maps. This patch
adds a boolean to convey the information to visitTypeEntry and
visitDerivedType that the pointee is a map definition and should be
treated as such.
It ressembles and works with commit 0d21c956a5c1 ("[BPF] Handle nested
wrapper structs in BPF map definition traversal (#144097)") which
focused on directly nested wrapper structs.
Before that patch, this ARRAY_OF_MAPS definition would lead to the BTF
information include the 'missing_type' as "FWD 'missing_type'
fwd_kind=struct":
struct missing_type { uint64_t foo; };
struct {
__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
[...]
__array(
values, struct {
[...]
__type(value, struct missing_type);
});
} map SEC(".maps");
Which lead to errors while trying to load the map:
libbpf: map 'outer_map.inner': can't determine value size for type [N]:
-22.
To solve this issue, users had to use the struct in a dummy variable or
in a dummy function for the BTF to be generated correctly [^2].
[^1]: https://lore.kernel.org/netdev/aH_cGvgC20iD8qs9@gmail.com/T/#u
[^2]:
https://github.com/cilium/ebpf/discussions/1658#discussioncomment-12491339
---------
Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>
Co-authored-by: Eduard Zingerman <eddyz87@gmail.com>
The object file format specific derived classes are used in context like MCStreamer and MCObjectTargetWriter where the type is statically known. We don't use isa/dyn_cast and we want to eliminate MCSection::SectionVariant in the base class.
In Aya/Rust, BPF map definitions are nested in two nested types:
* A struct representing the map type (e.g., `HashMap`, `RingBuf`) that
provides methods for interacting with the map type (e.g. `HashMap::get`,
`RingBuf::reserve`).
* An `UnsafeCell`, which informs the Rust compiler that the type is
thread-safe and can be safely mutated even as a global variable. The
kernel guarantees map operation safety.
This leads to a type hierarchy like:
```rust
pub struct HashMap<K, V, const M: usize, const F: usize = 0>(
core::cell::UnsafeCell<HashMapDef<K, V, M, F>>,
);
const BPF_MAP_TYPE_HASH: usize = 1;
pub struct HashMapDef<K, V, const M: usize, const F: usize = 0> {
r#type: *const [i32; BPF_MAP_TYPE_HASH],
key: *const K,
value: *const V,
max_entries: *const [i32; M],
map_flags: *const [i32; F],
}
```
Then used in the BPF program code as a global variable:
```rust
#[link_section = ".maps"]
static HASH_MAP: HashMap<u32, u32, 1337> = HashMap::new();
```
Which is an equivalent of the following BPF map definition in C:
```c
#define BPF_MAP_TYPE_HASH 1
struct {
int (*type)[BPF_MAP_TYPE_HASH];
typeof(int) *key;
typeof(int) *value;
int (*max_entries)[1337];
} map_1 __attribute__((section(".maps")));
```
Accessing the actual map definition requires traversing:
```
HASH_MAP -> __0 -> value
```
Previously, the BPF backend only visited the pointee types of the
outermost struct, and didn’t descend into inner wrappers. This caused
issues when the key/value types were custom structs:
```rust
// Define custom structs for key and values.
pub struct MyKey(u32);
pub struct MyValue(u32);
#[link_section = ".maps"]
#[export_name = "HASH_MAP"]
pub static HASH_MAP: HashMap<MyKey, MyValue, 10> = HashMap::new();
```
These types weren’t fully visited and appeared in BTF as forward
declarations:
```
#30: <FWD> 'MyKey' kind:struct
#31: <FWD> 'MyValue' kind:struct
```
The fix is to enhance `visitMapDefType` to recursively visit inner
composite members. If a member is a composite type (likely a wrapper),
it is now also visited using `visitMapDefType`, ensuring that the
pointee types of the innermost stuct members, like `MyKey` and
`MyValue`, are fully resolved in BTF.
With this fix, the correct BTF entries are emitted:
```
#6: <STRUCT> 'MyKey' sz:4 n:1
#00 '__0' off:0 --> [7]
#7: <INT> 'u32' bits:32 off:0
#8: <PTR> --> [9]
#9: <STRUCT> 'MyValue' sz:4 n:1
#00 '__0' off:0 --> [7]
```
Fixes: #143361
Currently, to avoid generating too much BTF types, for a struct type
like
```
struct foo {
int val;
struct bar *ptr;
};
```
if the BTF generation reaches 'struct foo', it will not generate actual
type for 'struct bar' and instead a forward decl is generated. The
'struct bar' is actual generated in BTF unless it is reached through a
non-struct pointer member.
Such a limitation forces bpf developer to hack and workaround this
problem. See [1] and [2]. For example in [1], we have
```
struct map_value {
struct prog_test_ref_kfunc *not_kptr;
struct prog_test_ref_kfunc __kptr *val;
struct node_data __kptr *node;
};
```
The BTF type for 'struct node_data' is not generated. Note that we have
a '__kptr' annotation. Similar problem for [2] with a '__uptr'
annotation. Note that the issue in [1] has been resolved later but the
hack in [2] is still needed.
This patch relaxed the struct type (with struct pointer member) BTF
generation if the struct pointer has a btf_type_tag annotation.
[1] https://lore.kernel.org/r/20230310230743.2320707-4-davemarchevsky@fb.com
[2] https://lore.kernel.org/r/20241023234759.860539-9-martin.lau@linux.dev
Currently, middle-end generates 'unreachable' insn if the compiler
feels the code is indeed unreachable or the code becomes invalid
due to some optimizaiton (e.g. code optimization with uninitialized
variables).
Right now BPF backend ignores 'unreachable' insn during selectiondag
lowering. For cases where 'unreachable' is due to invalid code
transformation, such a signal will be missed. Later on, users needs
some effort to debug it which impacts developer productivity.
This patch enabled selectiondag lowering for 'unreachable' insn.
Previous attempt ([1]) tries to have a backend IR pass to filter
out 'unreachable' insns in a number of cases. But such pattern
matching may misalign with future middle-end optimization with
'unreachable' insns.
This patch takes a different approach. The 'unreachable' insn is
lowered with special encoding in bpf object file and verifier
will do proper verification for the bpf prog. More specifically,
the 'unreachable' insn is replaced by a __bpf_trap() function.
This function will be a kfunc (in ".ksyms" section) with a weak
attribute, but does not have definition. The actual kfunc definition
is expected to be in kernel. The __bpf_trap() extern function
is also encoded in BTF. The name __bpf_trap() is chosen to satisfy
reserved identifier requirement.
Besides the uninitialized variable case, the builtin function
'__builtin_trap' can also generate kfunc __bpf_trap(). For example
in [3], we have
```
# define __bpf_unreachable() __builtin_trap()
```
If the compiler didn't remove __builtin_trap() during middle-end
optimization, compilation will fail.
With this patch, compilation will not fail and __builtin_trap()
is converted to __bpf_trap() kfunc. The eventual failure will be
in verifier instead of llvm compilation. To keep compilation
time failure, user can add an option like `-ftrap-function=<something>`.
I tested this patch on bpf selftests and all tests are passed.
I also tried original example in [2] and the code looks like below:
```
; {
0: bf 16 00 00 00 00 00 00 r6 = r1
; bpf_printk("Start");
1: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x0 ll
0000000000000008: R_BPF_64_64 .rodata
3: b4 02 00 00 06 00 00 00 w2 = 0x6
4: 85 00 00 00 06 00 00 00 call 0x6
; DEFINE_FUNC_CTX_POINTER(data)
5: 61 61 4c 00 00 00 00 00 w1 = *(u32 *)(r6 + 0x4c)
; bpf_printk("pre ipv6_hdrlen_offset");
6: 18 01 00 00 06 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x6 ll
0000000000000030: R_BPF_64_64 .rodata
8: b4 02 00 00 17 00 00 00 w2 = 0x17
9: 85 00 00 00 06 00 00 00 call 0x6
10: 85 10 00 00 ff ff ff ff call -0x1
0000000000000050: R_BPF_64_32 __bpf_trap
11: 95 00 00 00 00 00 00 00 exit
<END>
```
Eventually kernel verifier will emit the following logs:
```
10: (85) call __bpf_trap#74479
unexpected __bpf_trap() due to uninitialized variable?
```
In another internal sched-ext bpf prog, with the patch we have bpf code:
```
Disassembly of section .text:
0000000000000000 <scx_storage_init_single>:
; {
0: bc 13 00 00 00 00 00 00 w3 = w1
1: b4 01 00 00 00 00 00 00 w1 = 0x0
; const u32 zero = 0;
...
0000000000003a80 <create_dom>:
; {
1872: bc 16 00 00 00 00 00 00 w6 = w1
; bpf_printk("dom_id %d", dom_id);
1873: 18 01 00 00 3f 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x3f ll
0000000000003a88: R_BPF_64_64 .rodata
1875: b4 02 00 00 0a 00 00 00 w2 = 0xa
1876: bc 63 00 00 00 00 00 00 w3 = w6
1877: 85 00 00 00 06 00 00 00 call 0x6
; ret = scx_bpf_create_dsq(dom_id, 0);
1878: bc 61 00 00 00 00 00 00 w1 = w6
1879: b4 02 00 00 00 00 00 00 w2 = 0x0
1880: 85 10 00 00 ff ff ff ff call -0x1
0000000000003ac0: R_BPF_64_32 scx_bpf_create_dsq
; domc->node_cpumask = node_data[node_id];
1881: 85 10 00 00 ff ff ff ff call -0x1
0000000000003ac8: R_BPF_64_32 __bpf_trap
1882: 95 00 00 00 00 00 00 00 exit
<END>
```
The verifier can easily report the error too.
A bpf flag `-bpf-disable-trap-unreachable` is introduced to disable
trapping for 'unreachable' or __builtin_trap.
[1] https://github.com/llvm/llvm-project/pull/126858
[2] https://github.com/msune/clang_bpf/blob/main/Makefile#L3
[3] https://github.com/libbpf/libbpf/blob/master/src/bpf_helpers.h
Co-authored-by: Nikita Popov <github@npopov.com>