Row of the left matrix (yellow) meets column of the right matrix (blue). Multiply position by position, add up.
Row i of A = arrows leaving i (yellow). Column j of A = arrows entering j (blue). A product term is 1 only when an arrow leaves i and an arrow enters j through the same middle vertex, which is a two-step walk. The green walks on the graph are what the number counts.