Graph Attention

A graph convolution treats every neighbour as equally worth listening to. Graph attention learns the weights instead, so a node can lean on the two neighbours that matter and ignore the forty that do not. It is the same idea as attention in a transformer, restricted to the edges the graph actually has.

Attention here is masked by adjacency: scores are computed only for real edges, so cost scales with edges rather than with nodes squared. That restriction is the whole difference from a transformer over nodes, and it is why graph attention stays cheap on sparse graphs and why it inherits the graph's blind spots — a pair with no edge gets no score, however related they are.

α sub _uv  = softmax sub _u (LeakyReLU(a super ^T [Wh sub _v  ‖ Wh sub _u ])) over u ∈ N(v), then h sub _v ' = σ(Σ sub _u  α sub _uv  W h sub _u ). The softmax runs over the neighbourhood rather than the whole graph, so attention mass a node has to spend is fixed at one and divided among however many neighbours it has — more neighbours means less each, before any learning.