Can square matrices be replaced by n-gons?

A matrix is a rectangle of numbers. What if the numbers were laid out as a triangle, a pentagon or a hexagon? "Multiplying shapes" can mean three precise things, and each gives a different, sometimes surprising answer.

The common thread: the rectangle is what multiplication converges to, and other shapes earn their place only with a different multiplication. See Takeaways.

Overview

The page asks whether the rectangle of a matrix can be replaced by another shape. Sections 1–3 answer it mathematically. Sections 4–6 try shaped weights in language models, first one layer in a tiny model, then whole models, ending in a long series of experiments on a pyramid model with matrix products as its feed-forward layers.

1 · Shapes as nonzero patterns inside a matrix

Keep ordinary matrix multiplication, but allow nonzeros only inside a shape drawn on the N×N grid. The product of two such matrices has nonzeros where (x,z) can be reached through some middle index y with both (x,y) and (y,z) inside the shape. That is the relational composition R∘R. A shape is useful as a "matrix type" only if it is closed: the product of two such matrices has the same shape.

What survives multiplication

• Stable shapes (R∘R = R) have at most 5 corners. Any box whose row and column ranges overlap is stable, and stays stable when cut by the diagonal half-plane x ≤ y (or x ≥ y). That gives rectangles, triangles, trapezoids and pentagons, and that direction is proved. In a random search of 4,171 polygons, all 507 stable ones had only horizontal, vertical and diagonal edges. So no hexagon or larger n-gon is stable (numerical evidence for this direction).
• Closed but dying. A shape whose column range misses its row range multiplies to zero, and some shapes shrink into themselves. Those can be any n-gon, but their powers vanish (nilpotent).
• Multiplication squares things up. Under repeated products, every shape crossing the diagonal converges to its bounding rectangle. Band matrices, which are hexagons inside the square, fill the whole matrix after about 1/bandwidth products.
• The triangle is the classic win: closed, the building block of LU and Cholesky factorisations, and a product costs N³/6 instead of N³. But upper × lower is dense.
• A triangle is also how symmetric matrices are stored. Their ordinary product isn't symmetric, but the Jordan product (AB + BA)/2 and powers Aᵏ are. So triangle storage is closed under those, even though the triangle pattern of a symmetric matrix is the whole square.

2 · Shaped arrays multiplied by convolution

A non-rectangular array has no natural "row times column" product. It does have a natural product, though: treat it as a two-variable polynomial (or an image filter) and convolve. The shape of the result is the Minkowski sum P + Q, so a pentagon times a pentagon is a pentagon twice the size. The exact minimum number of multiplications is the number of lattice points in P + Q. By Pick's theorem that is 4|P| − B − 3, where B counts the lattice points on the boundary.

• Exact cost. The output coefficients are independent, so at least |P+Q| multiplications are needed. Evaluating both arrays at |P+Q| generic points, multiplying, and interpolating attains it. The catch is numerical: that interpolation is badly conditioned (condition numbers of 10⁶–10¹⁰ already for 15–25 points), which is why practice uses FFTs on boxes or hexagons.
• Boundary points are cheap. Lattice-aligned edges (triangle, square, hexagon) carry many lattice points. Pentagon, octagon and disk edges cut the lattice at awkward angles and carry few, so at equal size they cost about 4% more per coefficient.
• No gaps in 2D, gaps in 3D. For every 2D lattice polygon, (P∩Z²) + (P∩Z²) = (2P)∩Z² (0 failures in 300 random polygons). In 3D this fails: the double of the Reeve tetrahedron contains (1,1,1) and (1,1,2), which are not sums of two of its points.
• Regular polygons exist on a lattice only for n = 3, 4, 6 (the crystallographic restriction). Pentagons and octagons can only be approximated, which is why their boundaries are lattice-poor.
• Mixing shapes: triangle + reflected triangle = hexagon (correlating triangular arrays yields hexagons); square + diamond = octagon; pentagon + reflected pentagon = decagon.

3 · A polygon of matrices: trace(A₁A₂···A_k)

Put a matrix on each edge of a k-gon and multiply around the ring. This k-linear form is the k-gon tensor network. The triangle is ordinary matrix multiplication, trace(ABC), with Strassen's 7 products for 2×2. The square is the four-way form trace(ABCD) from the three-way project. Its rank N⁴ means no fused algorithm beats the naive one. The tensor rank is the number of products a fused algorithm needs. For 2×2 matrices (bond dimension 2):

• Even polygons give no shortcut. Splitting the ring into alternating vertices cuts every edge, and that flattening has rank 2^k, equal to the number of naive terms. So square, hexagon and octagon rings have rank exactly 2^k.
• Odd polygons do, because the alternating split must leave one edge uncut. Christandl–Zuiddam's tensor surgery turns Strassen's triangle into 2^k − 1 products for every odd k: 31 for the pentagon, 127 for the heptagon.
• The pentagon is open: 25 ≤ R ≤ 31. The lower bound comes from a Young flattening (Buhrman et al.), which also bounds the border rank. We attacked it numerically:
  • random restarts, and restarts from the exact 31-term decomposition with one term removed, all stall at error exactly 1;
  • a border-rank probe at 30 terms from there shows no divergence;
  • an exact GF(2) SAT search is described below;
  • a rotation-symmetric search (orbits of 5 rotated terms plus fixed terms, the form in which it finds Strassen's triangle) gives one lead. At 30 = 5·5 + 5 terms the error keeps falling, 0.044 → 0.025 over a long run, while the coefficients slowly grow (error ∝ norm^−2.1). That is the signature of an approximate (border) decomposition, which would beat the best known border bound of 31. It is not confirmed: double precision runs out long before the error could reach zero. The 29- and 31-term symmetric fits freeze at 0.041 and 0.0003.
  So the exact rank stays open. 31 has not been beaten exactly, and there is one unconfirmed numerical hint that 30 terms might work approximately.

4 · Triangle-shaped weights, swept three ways

Store weights on a triangular lattice: cells (i, j, k) with i + j + k = n − 1, about n²/2 of them. Each cell lies on three lines, at 0°, 60° and 120°. A matrix entry lies on only two: its row and its column. Sweeping each direction multiplies the two other coordinates' inputs and sums along the line. So one set of weights gives three outputs:

y₀[i] = Σ W·hj·hk   y₁[j] = Σ W·hi·hk   y₂[k] = Σ W·hi·hj

Together the three sweeps are exactly the gradient of the cubic energy E(h) = Σ W_ijk·h_i·h_j·h_k. That makes it a multiplicative layer, in the same family as the gated layers (GLU, SwiGLU) that modern language models use.

Does it help a language model?

A 2-layer, width-64 character-level transformer was trained on this repository's markdown (2.8 MB, 1,500 steps, CPU). Only the feed-forward block differed, and every variant had about 65.5k feed-forward weights. Lower is better:

• It works: both triangle layers beat the standard MLP at the same number of weights.
• It isn't the best: the standard gated layer, SwiGLU, wins.
• Three sweeps don't help quality per weight. Each extra direction needs its own output weights, so at a fixed budget the three-sweep triangle must be smaller (n = 106 against 158 for one sweep).
• What three sweeps buy is reuse: each triangle weight feeds 6 multiplications per read, against 1 for an ordinary matrix weight. Generating text one token at a time is limited by memory bandwidth, so that is where this could pay off. Testing it needs a fused GPU kernel; this CPU version runs about 25× slower per step than the MLP, an implementation artefact.
• Triangles already work elsewhere in big models, over tokens or pairs rather than weights. AlphaFold's triangle updates refresh each residue pair from the other two edges of every triangle containing it, swept in two directions. 2-simplicial attention (Meta, 2025) attends over triangles of tokens and improves token efficiency on maths, code and reasoning.
• Caveat: tiny model, short run, CPU. This is a smoke test, not a scaling result.

In 3D: a cube and a tetrahedron

A cube of weights W[i, j, k] is a full 3-way array. Each weight sits on 3 lines, one per axis, and the triangle above is just its diagonal slice i + j + k = n − 1. A tetrahedron (cells with i + j + k + l = n − 1) has 4 face directions. Each sweep collapses a triangular slice and multiplies the other three inputs, giving a 4th-order (quartic) interaction. In general a d-dimensional simplex has d + 1 sweeps and d(d + 1) multiplications per weight read.

In the same tiny language model, at the same 65.5k feed-forward weights (cube n = 29, tetrahedron n = 46):

• The cube is the best shaped layer: second only to SwiGLU at 1,500 steps, and ahead of its own diagonal slice, the triangle, at the same number of weights. It also runs as plain dense array operations, so it was 8× faster per step than the triangle here.
• The tetrahedron has the most reuse (12 multiplications per weight read) but was too slow on this CPU to train long enough to judge. At 500 steps it ties everything.

Fewer bits?

Does a shaped weight array, used three ways, keep its quality with fewer bits per weight? The trained models' feed-forward weights were rounded to b bits after training (symmetric, per tensor, round to nearest) and the language model re-evaluated. "Gated" is the SwiGLU-style feed-forward block: Swish-gated linear unit, then a projection.

• Rounding everything to the same bit width gives no bonus. At 3–4 bits the triangle and cube degrade slightly faster than the MLP and the gated block.
• But the three-way weights themselves are robust: the cube's 49k three-way weights at 2 bits cost less than a single 33k MLP matrix at 2 bits.
• The weak point is the input projection. Its rounding errors get multiplied together inside the products.
• So mixed precision wins. With the three-way weights at 3 bits and the input projection kept precise, the cube needs fewer bits for the same quality, beating the gated block by 0.07 bits/char at about 200k stored bits. The triangle doesn't gain: its wide output projection dominates the storage.

5 · Hexagons: wrap around or not, and hexagonal attention

Multiply two hexagonal arrays by convolution. With wrap-around (hexagons tiling a torus) every output cell receives exactly the same number of multiplications, |P|, just like an ordinary cyclic product. Without wrap-around the count is the overlap of two hexagons: |P| at the centre, falling to 1 at the corners.

Does the centre learn faster?

Yes. Under gradient descent an output's learning speed is its neural-tangent-kernel diagonal, Σ_p b²_s−p + a²_p, which grows with the number of products landing on it. Both factors were trained (y = a ∗ b, random targets, 12 seeds, hexagons of radius 4):

• Without wrap-around the centre learns about 30× faster than the corners (learning speed 41 against 2; error removed 9.4 against 0.29). The edges are under-trained, a built-in curriculum from the inside out.
• With wrap-around every output learns at the same speed, like an ordinary dense or cyclic product.
• The input cells, by contrast, are uniform either way: each one meets |P| outputs. Only the outputs differ.

A hexagonal attention or convolution mask

The idea: let each cell attend strongly to its 6 neighbours, skip the next ring, and attend to a few random far cells. That is a hexagonal cousin of BigBird's local + random sparse attention.

• Random far links are cheap and powerful. Ring 1 alone needs 16 layers to connect a radius-8 map. Adding 3 random far cells per cell needs only 5, with 9.5 links per cell against 16.7 for a dense two-ring mask that still needs 8. It's the small-world effect.
• Skipping ring 2 works like a dilated kernel: ring 1 + ring 3 connects the map in 6 layers.
• Hexagonal blurs are rounder. Repeating a 7-cell hexagonal blur gives a spread that is exactly isotropic in its 4th moments (max/min over directions 1.000 at every step). Square blurs are only isotropic in their 2nd moments: their 4th-moment ratio is 1.29 after 2 steps (4 neighbours) and still 1.04 after 10. Six-fold symmetry cancels the 4-fold term that squares keep.

In a language model: does the non-uniformity help?

Arrange the hidden units of the feed-forward block on a hexagon and let the block convolve two projections of the input across it: y = C·((A·x) ∗ (B·x)). Without wrap-around the output hexagon has twice the radius and is centre-heavy; with wrap-around every output gets the same number of products. This is computed exactly with an FFT, because a hexagon of radius R on a torus is the cyclic group Z_N, N = 3R² + 3R + 1, via φ(q, r) = ((R+1)·q − R·r) mod N. The question is not whether it saves operations, but whether the shape gives the model a property it can use.

• The centre-heavy layer learns an importance order on its own. Trimming the outer rings hurts 2–3× less than dropping random cells (82% of cells kept: +0.08 against +0.23). The uniform, wrapped layer has no such order: outer and random drops cost the same. That gives a ready-made dial for trimming compute at inference, outer rings first. It isn't free (trimming a third costs +0.23), but it's much gentler than trimming at random.
• More precision at the centre does not help. With the output weights at a 3-bit average: uniform +0.18, more bits at the centre +0.34, more bits at the edge +0.49. Squeezing any ring down to 2 bits costs more than the others gain.
• Training pushes back against the non-uniformity. Edge activations are about 5× weaker than the centre's, so the edge output weights grow larger and move about 70% further during training. The compensation is partial: the centre still contributes about 3× more, which is why the outer rings are cheaper to trim.
• Next: train with random outer-ring dropout (Matryoshka-style) to sharpen the order, and use it for elastic width or early exit.

Where this matters: hexagonal grids are standard in geospatial indexing (Uber's H3), board and strategy game maps, telescope and detector arrays, and hexagonal CNNs (HexagDLy). A hexagonal local + skip + random mask is a natural sparse attention for such data. Choosing wrap-around or not decides whether the border learns as fast as the middle.

6 · A whole language model in a shape

The experiments above swap one layer. shape-llm builds a whole small character-level model around a shape. Every token's hidden state is a hexagon of feature cells, shrinking through the network like an image pyramid.

• Each block has attention across tokens, then a feed-forward that is either hexagonal (each cell mixes with its 6 neighbours) or the triangle layer of section 4.
• Between levels, 3 neighbouring cells become 1. The small triangles {(q, r), (q+1, r), (q, r+1)}, one per point with q − r divisible by 3, tile the grid exactly. Their positions form a coarser hexagonal grid, rotated 30° and scaled √3. Cropping to radius 6 and 3 drops the outer band of cells.
• The baseline is an ordinary transformer with the same depth and parameter count.

Hexagonal feed-forward: clearly worse

• It learns the centre-first importance order again. At the radius-3 level, which the output reads in full, trimming the outer ring first costs +0.18 against +0.32 for random cells, and the edge activations are about 35% weaker. At the larger levels the crop builds part of that order in.

Turning the centre down

Centre outputs collect many more products, so perhaps more noise. The non-wrapped hexagonal layer of section 5 was scaled so the corners keep weight 1 and the centre gets less:

No sign that the centre's larger sums were harmful noise: 1/√ balances the layer at no cost, and 1/fan-in overshoots.

Triangle feed-forward: almost a standard transformer

Same pyramid and bottlenecks, but every block uses the triangle layer, swept at 0°, 60° and 120°. Without wrap-around the line at index i holds n − i weights, so long lines are the triangle's "centre". With wrap-around the triangle closes into a torus and every line is equally long. "Centre lowered" scales each output by (products)^−½; "reversed" by (products ÷ max)^½; "average 1" renormalises so only the balance changes, not the overall size.

Which triangle is which. The names in the tables and the chart combine two choices: wrap or not, and how the signal is spread from the long "centre" lines to the short edge lines. The weights shown are for the smallest triangle (side 50); the larger ones are similar.

Longer run: every triangle variant, 4,000 steps

The same full pyramid model, trained 4× longer with 3 seeds per variant, all on the same text. The letters are the two choices: A1 no wrap, A2 wrap; B1 full signal in the centre and lower at the edges, B2 low in the centre and full at the edges, B3 full everywhere. With wrap every line is equally long, so there is no centre: A2 + B1 is the same model as A2 + B3, and A2 + B2 only turns every line down by one factor.

• The triangle rescues the pyramid. It is 0.18 bits/char better than the hexagonal feed-forward and within 0.02 of a standard transformer at 1,000 steps (but see the longer run below). So the pyramid and its 3 → 1 bottlenecks were not the problem; the hexagon's local 6-neighbour mixing was.
• In the triangle, the centre is signal. Its long lines carry the important features: dropping 20% of outputs costs +0.16 longest-first, +0.10 random, +0.05 shortest-first. Turning them down hurts, and training grows their output weights back. Turning them up (reversed) is free or slightly helpful. The asymmetry is the opposite of the hexagon, where lowering the centre was neutral.
• Wrap looked like a wash at 1,000 steps: slightly better in the single layer, slightly worse in the full model, both within seed noise. The longer run below changes that.
• The curves: the transformer learns fastest early (3.79 at step 400 against 3.87–3.90 for the triangles), and the triangles catch up by step 1,000. In batch 3 the reversed triangle (average 1) edged ahead at step 1,000 (3.391 vs 3.404), but its seeds were 0.19 apart, so that crossing was never a result.
• At 4,000 steps the transformer still wins, but the wrapped triangle is right behind it. The transformer ends at 2.726 and the wrapped triangle at 2.739, only 0.013 apart, and their seed ranges overlap (2.713–2.737 against 2.732–2.751). Three seeds cannot separate them.
• Wrap is the best triangle, and this time the gap is real. Wrapped 2.739 against 2.778 without wrap, and the ranges do not overlap (worst wrapped seed 2.751, best unwrapped 2.761). One caveat: with the same side length the wrapped triangle holds n² weights instead of about n²/2, so it does twice the three-way products per step. That costs only 0.6% more parameters, but about 1.5× the time per step.
• The centre-versus-edge question, settled. Without wrap, raising the centre (B1) and leaving everything at full weight (B3) end level: 2.770 and 2.773 against 2.778, all within their seed ranges. Lowering the centre (B2) is the only clear loss: 2.789 with average 1, 2.817 without. So the long lines carry signal and should not be turned down, but boosting them does not buy anything measurable either.
• The earlier crossing was noise. At 1,000 steps the reversed triangle (average 1) edged ahead of the transformer. At 4,000 steps it is 0.044 behind, with non-overlapping seed ranges (worst transformer 2.737, best reversed 2.754). The triangles lead only at step 400, fall up to 0.08 behind around steps 1,200–2,000, then close in as training settles.
• Cost. On the CPU a triangle step takes about 4× as long as a transformer step without wrap and about 6× with wrap, because the three sweeps are many small gathers instead of one large matrix multiply.

What the longer runs revealed

1. Uniform beats every centre/edge weighting. The best triangle is the one with no centre at all (wrapped). Every no-wrap profile ranks below it: full centre, flat or lowered centre. The hexagon said the same thing in section 5: its wrapped single layer also beat the unwrapped one (3.289 against 3.324, seed ranges apart). In both shapes, the version where every output collects the same number of products wins. That is also the property a dense matrix has.
2. The 1,000-step runs got the ranking wrong twice. There, wrap finished 0.034 behind no wrap, and the reversed triangle looked ahead of the transformer. In the 4,000-step runs wrap pulls ahead from step 800 and the lead grows (0.009 at step 800, 0.031 at step 1,600, about 0.04 from step 2,400 on). The reversed triangle ends 0.044 behind the transformer. Seed spreads also shrank, from 0.19 at the end of a 1,000-step run to at most 0.045 at the end of a 4,000-step run. Short runs with 2 seeds were not enough to rank variants 0.03 apart.
3. The triangles start fast, fall behind, then close in. At step 400 every triangle except the two centre-lowered ones is slightly ahead of the transformer. By step 1,200 they are 0.07–0.09 behind (0.12 with the centre lowered). From there the gap shrinks steadily as the learning rate anneals: the wrapped triangle goes from 0.073 behind to 0.013. In the last 1,600 steps the triangles improve by 0.16–0.17 bits/char against 0.14 for the transformer, while in steps 400–2,000 the transformer improved more (0.99 against 0.87–0.95). A guess, not tested: the three-way products train less well at a high learning rate, so a lower peak rate or a longer schedule could help them.
4. The shape is mostly wiring, not weights. The triangle's own weights are only 14,150 numbers, 0.6% of the model (27,900 with wrap). Almost all parameters sit in the projections into and out of the triangle. Yet switching the wiring from "no wrap" to "wrap" moves the loss by 0.04, more than any reweighting of the centre and edges. What matters is which features get multiplied together.
5. Renormalising made a bad idea consistent. The centre-lowered triangle rescaled to average 1 had by far the tightest seeds (2.787–2.791), while the same profile without rescaling had the widest (2.787–2.832). Rescaling did not make it good, only predictable. This is one observation on 3 seeds.
6. Per unit of time, the transformer wins easily. Matched on parameters (the triangles even have 3–4% more), the triangle is close to the transformer. But each step is about 4–6× slower on a CPU, so at equal training time the transformer is far ahead. The triangle's case would rest on a fused gather–multiply kernel, which is the same open question as in section 4.

If you try triangle layers: wrap them, leave the weights unscaled, keep all three sweeps in every step and do not expect a fourth direction to help (see below), compare against a dense baseline at 4,000+ steps with 3 or more seeds, and test the learning rate before reading anything into gaps of 0.02. The fair next test of wrap is a smaller wrapped triangle doing the same number of products as the unwrapped one, so that wrap and extra work are separated.

Two of three sweeps per step

An idea to make the triangle cheaper: every training step uses only two of the three directions, and the pair rotates so that all directions get exercised.

angles = [0°, 60°, 120°]
a = angles[i % 3]            # i = training step, or a random number
b = angles[(i + 1) % 3]
sweep only a and b; skip the third direction

Tested on the wrapped and the unwrapped triangle, with the cyclic rule above and with a randomly chosen skipped direction. All layers skip the same direction on a given step. The two remaining sweeps are scaled by 3/2 during training (as dropout does), so the model can be evaluated with all three. 4,000 steps, 3 seeds, same text as the longer run.

• It costs 0.10–0.13 bits/char. That is more than the whole spread between the best and worst all-three-sweeps triangles (0.078), and the seed ranges are far apart (worst all-3 wrapped seed 2.751, best 2-of-3 wrapped seed 2.839).
• The gap grows during training, it does not close. For the wrapped triangle with the cyclic rule: 0.03 behind at step 400, 0.06 at step 800, 0.09 at step 2,000, 0.13 at the end. So it does not behave like a regulariser that pays off later.
• Cyclic or random makes no difference. Random is 0.012 better with wrap and 0.012 worse without, both inside the seed ranges.
• The rotation does what it was meant to: no direction is left behind. Evaluating with any two of the three sweeps gives nearly the same loss whichever direction is skipped (the three pairs differ by 0.03 at most). But two sweeps are always about 0.03 worse than all three, so cheaper inference is not free either.
• The saving is small. Skipping a third of the sweeps makes a training step only about 15% faster (0.66 → 0.56 s without wrap, 1.14 → 0.97 s with wrap, one CPU thread), because attention and the projections take most of the time. At equal training time it still loses: the all-three wrapped triangle is already at 2.780 after 3,200 steps, well ahead of 2.866.
• Why, probably: this is dropout on a third of the layer's outputs, applied to a small model trained briefly. Such a model usually has no spare capacity to spend on noise (the gap between training and validation loss was not measured). The loss of 0.03 when evaluating with two sweeps also shows that each direction carries information the other two do not.
• Not tested: skipping a direction less often, letting each layer skip a different direction, or skipping without the 3/2 rescaling.

A square with four directions (4 of 4)

The hypothesis: two of three sweeps was worse than three of three, so a fourth direction might do better still, and beat the standard transformer, whose matrices only pair rows with columns.

The layer keeps its weights on an n × n grid that wraps around, and sweeps it at 0°, 45°, 90° and 135°: rows, columns and both diagonals. With n odd, every cell lies on exactly one line of each direction. The first three directions are exactly the wrapped triangle of the sections above, so this is the wrapped triangle plus the second diagonal.

Each cell sits on four lines, one per direction. For the output of one of its lines, a cell contributes its weight times the features of its other three lines. Those three can be combined in two ways, and both were trained:

• One product across all four directions: the three features are multiplied together, so every product ties all four directions together (three give the features, the fourth receives the result). This is the direct extension of the triangle. In this view a plain matrix is the 2-direction case and the triangle the 3-direction case.
• Two directions at a time: the three features are multiplied in pairs and the pairs are added. This keeps the products two-way, as in the triangle, and changes only the number of directions.

• The hypothesis does not hold here. The best square ends 0.028 bits/char behind the transformer, and their seed ranges do not overlap (worst transformer seed 2.737, best square seed 2.747).
• A fourth direction is no better than three. The square is 0.015 behind the wrapped triangle. Their seed ranges touch (2.732–2.751 against 2.747–2.762), so the honest reading is "no gain, perhaps a small loss".
• Why "2 of 3 is worse" did not predict this. Skipping a sweep removes part of a model that was built for three directions. Adding a direction at a fixed parameter budget has a price: the grid shrinks from side 50 to 41 at the first level, so every direction has 18% fewer lines to work with. More directions on a smaller grid came out about level with fewer directions on a larger one.
• One product across all four directions is the better square: 2.754 against 2.781 for two at a time, and it is also faster (1.05 against 1.25 s per step). At step 400 it shared the best loss of all models (3.912, level with the reversed triangle), before falling behind the transformer like the triangles did.
• Not tested: a square with the triangle's side lengths (about 13% more parameters than the wrapped triangle), longer training, and a learning rate tuned for the three-way products.

What the square revealed

Is the fourth direction actually used? In each trained model, one direction's outputs were switched off in every layer (at evaluation only), and the rise in loss measured.

1. The fourth direction is fully used, but it divides the work instead of adding to it. Switching off any one of the square's four directions costs 0.23–0.25 bits/char, and the added diagonal (0.23) counts about as much as the other three. In the triangle each of the three directions costs 0.40. So each direction matters less in the square: the same job is shared by four directions instead of three, and the final loss is no better.
2. Removing and adding are not mirror images. Taking a direction away from a model built for three costs 0.10–0.13 even when the model is trained for it. Adding a fourth gains nothing. "Fewer is worse" did not imply "more is better".
3. At equal parameters, each added direction cost a little. Transformer 2.726, three directions 2.739, four directions 2.754: about 0.014 per added direction, the opposite of the hypothesis. Each single difference is close to the seed noise, and the transformer is not literally the two-direction member of the family (its layers use an activation function instead of products), so read this as a trend, not a law.
4. Uniform layouts take the top three places. Transformer, wrapped triangle, wrapped square with one product. Every non-wrapped triangle ranks below that square. Whether every output collects the same number of products mattered more than how many directions are swept.
5. The bolder product trained better. Multiplying three features at once could have been unstable. Instead it beat the cautious two-at-a-time version (2.754 against 2.781), with tighter seeds (spread 0.015 against 0.050).
6. The fourth sweep was free in time. The square with one product takes 1.05 s per step against 1.14 s for the wrapped triangle, because its smaller grid (41 × 41 instead of 50 × 50 at the first level) more than pays for the extra sweep.
7. Same learning pattern as the triangles. A fast start (level with the best model at step 400), then 0.07 behind the transformer around step 1,200–1,600, closing to 0.028 at the end.

All 4,000-step models side by side

All thirteen models trained for 4,000 steps on the same text, ranked by their final loss. The transformer is first, the wrapped triangle second and the square with one product across all four directions third. The two-of-three models are worse than every model that sweeps all its directions.

Time against loss. Seconds per training step against final loss, for the models with a clean timing.

A square matrix with the usual product, and a rotating offset

A different kind of layer: no sweeps, just ordinary matrix multiplication. Each feed-forward layer turns a token's features into a small square matrix M, computes the usual product M·M, and maps the result back. In the product, the destination cell [x, y] is row y times column x.

In blocks 1–2 the row and the column are taken from neighbours instead, and which neighbours depends on the training iteration. The destination cell itself is not moved:

xoffset = (iteration >> 1) & 1
yoffset = iteration & 1
row = matrix[(y + yoffset) % height]
col = matrix[(x + xoffset) % width]
dest_cell[x, y] = row · col

The matrices are 10 × 10 in blocks 1–2, 11 × 11 in blocks 3–4 and 13 × 13 in blocks 5–6, which gives 2.27M parameters (transformer: 2.25M). Blocks 3–6 use no offset, and validation uses no offset. One run of 1,000 iterations each, trained one after the other on the same text with the same seed, plus the same model with no offset at all. The first run used a mistyped formula, xoffset = (iteration*2) & 2, which only alternates between two states: no offset, and row y + 1 with column x + 2. It is kept as a second column. A third rule, xoffset = yoffset = iteration & 1, moves the row and the column together: no offset, then row y + 1 with column x + 1. A fourth does the same with a shift of 2: row y + 2 with column x + 2. A fifth cycles through three states: no offset, (x+1, y+1), (x+2, y+2). In the table, a column named (x+a, y+b) is a two-state rule: no offset on even iterations, that offset on odd ones.

• All five offset rules end ahead of the transformer: 3.258, 3.259, 3.275, 3.290 and 3.305, against 3.350. No other shape layer on this page has led the transformer by this much.
• Without the offset the same model is clearly worse: 3.482. That is 0.13 behind the transformer and 0.18–0.22 behind the offset versions. All these runs start from identical weights and see identical batches, so the offset is the only difference between them. An earlier version of this page guessed that the gain came from the matrix product itself. These runs say the opposite: the offset is what makes the difference. The part “What the alternating models actually learned” below shows how: the offset does not make the layer work, it makes the model stop relying on it.
• What the offset does. It does not change the product. The cell [x, y] receives the value that the plain product would put at [x + xoffset, y + yoffset], so the whole result is shifted before it is read out. The read-out is therefore trained on shifted copies of the same result, and has to work for all of them.
• No rule has beaten the first one. (x+2, y+2) ties with (x+2, y+1), at 3.259 and 3.258. The three-state cycle (0,0), (x+1, y+1), (x+2, y+2) lands at 3.290, between its two ingredients: the (x+1, y+1) rule at 3.305 and the (x+2, y+2) rule at 3.259. So far the two-state rules with a shift of 2 are best and the ones built from a shift of 1 are behind, which hints that the size of the shift matters more than the number of states. The whole spread is 0.05, though, which a single seed cannot resolve. What the five rules share matters more: every one of them beats the layer without an offset by a wide margin.
• It is one seed each, at 1,000 iterations. Seeds have differed by up to 0.19 at this length, and the short runs above ranked variants wrongly twice. A gap of about 0.2 between paired runs is the largest effect on this page, but it needs more seeds and a 4,000-step run before it counts as a result.
• The model does not depend on which state it is in. Evaluated in each of the other three states of the four-state cycle, the loss is 3.277, 3.277 and 3.281, against 3.275 with no offset. The (x+1, y+1) rule gives 3.304 in its shifted state against 3.305, and the (x+2, y+2) rule 3.265 against 3.259. The three-state rule gives 3.293 and 3.296 in its two shifted states against 3.290.
• It is cheap. A step takes 0.12–0.13 s against 0.058 s for the transformer: twice as slow, but about five times faster than the triangle layers, because a matrix product is one dense operation.

A staircase instead of straight rows and columns

The offsets above only move the result. This idea changes the product itself. In a matrix product, a destination cell is a sum along its row and along its column, both straight lines. Here both lines become staircases: two cells along, one across, two along, one across. That is a slope of 1 in 2, an angle of 26.565°.

The tilt alternates: +26.5° on even iterations, −26.5° on odd ones. The row path and the column path turn by the same angle, so they stay at right angles to each other. Everything wraps around at the edges.

s = +1 if iteration % 2 == 0 else -1
f(k) = k // 2                      # 2 cells along, 1 across

# through the output cell: the row path is at row y when k == x, the column path at column x when k == y
dest_cell[x, y] = sum over k of
    matrix[(y + s*(f(k) - f(x))) % height][k]  *  matrix[k][(x - s*(f(k) - f(y))) % width]

# from the edge (the first runs): the paths start at row y and column x
dest_cell[x, y] = sum over k of
    matrix[(y + s*f(k)) % height][k]  *  matrix[k][(x - s*f(k)) % width]

The first table and findings are for the paths from the edge; the version through the output cell follows below. As with the offsets, the staircase is used in blocks 1–2 and blocks 3–6 keep the plain product. One run, seed 0, 1,000 iterations, same text and same starting weights as the runs above. A second run keeps the tilt fixed at +26.5° on every iteration, to separate the bent paths from the alternation. A third cycles through three states: −26.5°, 0° (straight rows and columns) and +26.5°. A fourth uses a shallower staircase, 3 cells along and 1 across, which is ±18.435°. A fifth keeps that shallower tilt fixed at +18.4°. A sixth is steeper: 3 cells along, then 2 across, which is ±33.69°. Its path skips a cell each time it steps, so it still visits one cell per column.

• The staircase ends at 3.292: ahead of the transformer (3.350) and far ahead of the plain product (3.482), but behind the best offset rules (3.258 and 3.259). It sits inside the band of the offset rules, 3.258 to 3.305.
• Both tilts work equally well. Evaluated with the −26.5° tilt the loss is 3.288, against 3.292 with +26.5°.
• With a fixed tilt the staircase loses most of its advantage: 3.435. That is 0.14 worse than the alternating staircase, behind the transformer, and only 0.05 better than the plain product (3.482). Same paths, same weights at the start, same batches; the only difference is whether the tilt alternates.
• So the alternation helps more than the shape of the path. The two versions that never change sit at 3.44 and 3.48. The versions that alternate between two states land between 3.26 and 3.31, whether they shift the result or bend the paths. Bending the paths by itself is worth about 0.05, too little to tell apart from noise on one seed.
• From the edge, three states are worse than two: 3.388. Adding the straight product as a middle state costs 0.10 against the two-state staircase and puts the model behind the transformer (3.350), though still ahead of the fixed versions. This did not happen with the offsets, where the three-state rule (3.290) was level with the two-state ones. One possible reason: an offset only moves the same result around, while each tilt is a different calculation, so with three tilts each one is trained on only a third of the iterations. That is a guess from one seed.
• A shallower staircase helps less: 3.359 at ±18.4°, against 3.292 at ±26.5°. It ends just behind the transformer (3.350).
• The fixed +18.4° staircase ends at 3.459. That completes a small table of its own:

  tilt	fixed	alternating ±	gain from alternating
  0° (plain product)	3.482	–	–
  18.4° (3 along, 1 across)	3.459	3.359	0.10
  26.5° (2 along, 1 across)	3.435	3.292	0.14
  33.7° (3 along, 2 across)	not run	3.303	–

  Two things grow with the tilt. A fixed tilt helps a little by itself (about 0.02 from one tilt to the next, too small to trust on one seed). Alternating helps much more, and more at the steeper tilt. The offsets showed the same trend, where a shift of 2 beat a shift of 1. So a larger change between the states seems to help more, up to a point.
• Steeper than 26.5° brings nothing more: ±33.7° ends at 3.303, level with ±26.5° at 3.292 (0.011 apart, inside the noise). The gain grows from 18.4° to 26.5° and then flattens. This staircase is also a different kind of path, because it jumps over a cell at every step. Both tilts score the same, 3.303 and 3.302.
• Again the model does not care which state it is in. The three-state model gives 3.388, 3.390 and 3.391 in its three states, and the ±18.4° model 3.359 and 3.360.
• It is one seed at 1,000 iterations, so differences of a few hundredths between the alternating versions mean nothing yet.

Staircases through the output cell

In the runs above each path starts at the edge of the matrix, so it usually does not touch the cell it computes. Here every path is shifted so that it is the staircase of its family that passes through the output cell, as in the first two grids of the diagram above. All six staircase variants were trained again this way: same seed, same starting weights, same text, one run each.

• Through the output cell is better in all six pairs. The improvement ranges from 0.018 to 0.127. Any single gap of 0.02 would mean nothing on one seed, but six paired runs all moving the same way is the most consistent signal in these staircase experiments.
• The three-state staircase changes the most: 3.388 → 3.262. From the edge it was behind the transformer. Through the output cell it is the best staircase, and level with the best offset rule (3.258). The earlier finding that three states are worse than two holds only for paths from the edge.
• The two-state staircases gain a little: ±26.5° goes from 3.292 to 3.270, ±33.7° from 3.303 to 3.285, ±18.4° from 3.359 to 3.332. Their order stays the same, with 26.5° best.
• The fixed staircases gain more: +26.5° goes from 3.435 to 3.369 and +18.4° from 3.459 to 3.402. A fixed +26.5° staircase through the output cell is now within 0.02 of the transformer, where the plain product is 0.13 behind. So with the paths through the cell, the bent path helps by itself. Alternating still adds to it: 0.10 at 26.5° and 0.07 at 18.4°.
• A cost that did not hurt. Shifting every path through its own cell means some pairs of staircases are used by two output cells and others by none. In the 10 × 10 matrix only 76 of the 100 results are distinct (86 at 18.4°). The models are better all the same.
• Why: the models with paths through the output cell rely even less on the feed-forward of blocks 1–2 than the ones from the edge (for the three-state staircase, 0.002 against 0.07). See “What the alternating models actually learned” below.
• Still one seed at 1,000 iterations. The direction is consistent; the sizes are not established.

Fewer multiplications: a queens mask

The hypothesis: more multiplications did not help, so perhaps fewer do. A regular matrix product multiplies every row with every column. Here only N cells of the N × N matrix take part. This is no longer a matrix multiplication. The question is whether such a cheap operation is good enough as the non-linear step of the layer, the job an activation function normally does.

The N cells are picked as in the eight queens puzzle: one cell in every row and every column, and no two on the same diagonal. A new placement is drawn at random for every layer on every iteration, from a pool of 200 that covers every cell of the matrix, so over time every cell contributes.

How the picked cells are used was open, so both readings were trained. With p[y] the picked column of row y:

• Mask on both sides (the strict reading): the masked matrix is multiplied by itself. Only N products remain, M[y, p[y]] · M[p[y], p[p[y]]], and only N cells of the result are non-zero.
• Mask on one side: the masked matrix is multiplied by the full one. Every cell [x, y] gets a single product, M[y, p[y]] · M[p[y], x], so N² products in total.

• With a new random placement every iteration, fewer multiplications did not help. Against the full product without offset (3.482), the queens mask costs 0.27 bits/char on both sides (3.755) and 0.42 on one side (3.908). Both are far behind the transformer (3.350).
• With the same placement always, fewer multiplications are nearly free. The fixed mask on one side ends at 3.518, 0.036 behind the full product with a tenth of its multiplications, which is close to the noise. On both sides, with a hundredth of them, it ends at 3.564, 0.08 behind. Fixing the placement improved the two versions by 0.19 and 0.39.
• So the problem was the randomness, not the sparsity. The same mask, kept fixed, turns the worst model of these runs into one level with the plain matrix product. It matches the local product, which also uses a fixed subset of the cells and also matched the full product.
• With a fixed placement, one side is the better version. It fills every cell of the result, and now each cell means the same thing on every iteration. With a random placement the order was the other way round.
• The fixed models depend on their placement. Evaluated with a placement they never saw, they collapse: 4.29 to 4.33 on both sides, 5.18 to 5.37 on one side. The random-placement models do not care which placement they get, because they learned to ignore these layers.
• The saving is small. A step gets 27% faster (0.13 → 0.095 s), because the projections and attention dominate, not the product. Going from 1,000 multiplications to 10 buys little time.
• With random placements the sparser version is the better one. Ten products beat a hundred. With the mask on one side every cell is filled, but what each cell means changes with every new placement. With the mask on both sides only the N picked values matter.
• The random-placement models do not depend on the particular placement. Evaluated with three fixed placements instead of random ones, the loss stays within 0.002 (both sides) and 0.01 (one side).
• Why: a new random mask on every iteration, in every block, taught these models to ignore all of their feed-forward layers, including the ones in blocks 3–6 that the model needs (measured below, in “What the alternating models actually learned”). A fixed mask, which makes the layer a consistent function, fixes most of this (above).
• A contrast worth noting: the rotating offset also rearranges the layer's output from one iteration to the next, yet it helped by about 0.2 while the random queens mask hurt by 0.27. The difference is where it is applied: the offset disturbs only blocks 1–2, the queens mask all six.
• All of this is one seed at 1,000 iterations, so only the large gaps should be trusted: the 0.19 to 0.39 from fixing the placement, and the 0.27 to 0.42 from a random one. All of these masks are applied in every block; none of them comes near the transformer.

What the alternating models actually learned

Every model above that alternates between states beats the plain matrix product. The trained models were opened up to find out why. No new training was needed for the three measurements; the fourth item is one control run.

1. The alternating models have switched the layer off. In the plain model, switching off the feed-forward of blocks 1–2 costs 0.405 bits/char. In the alternating models it costs between 0.002 and 0.14, and the layer's output is two to four times smaller. The offsets and staircases were applied to exactly those two blocks. A layer whose output keeps being rearranged is one the rest of the model learns not to depend on.
2. Removing the layer outright is better than any of them: 3.220. One control run with no feed-forward at all in blocks 1–2 (same seed, same starting weights, same text) beats the best offset rule (3.258), the best staircase (3.262) and the transformer (3.350). It is the best 1,000-iteration model on this page. Each block has two halves, attention and feed-forward. Only the feed-forward half of blocks 1 and 2 is removed. The model still has six blocks and six attention layers, but four feed-forward layers instead of six, and 17% fewer parameters in use (1.89M against 2.27M; the transformer has 2.25M). So it is a smaller model that does better, not a like-for-like swap.
3. The transformer does not gain from the same removal: 3.378. The standard transformer with the feed-forward half of its blocks 1 and 2 removed (same seed, same starting weights, 1.70M parameters in use instead of 2.25M) ends 0.028 behind the full transformer (3.350). So this is not a general rule that early feed-forward layers are useless. An ordinary feed-forward layer in the first two blocks does a little good; a matrix-product layer there does harm.

   feed-forward in blocks 1–2	matrix-product model	standard transformer
   present	3.482	3.350
   removed	3.220	3.378
   effect of removing it	0.262 better	0.028 worse

   With the two layers removed from both, the matrix-product model is 0.158 ahead of the transformer (3.220 against 3.378) at a similar size (1.89M against 1.70M parameters). The two models differ in more than the feed-forward, though: the matrix-product model also has the pyramid, its bottlenecks and a different attention.
4. So the offsets and staircases helped by taking something away, not by adding something. In this model the matrix-product feed-forward in the first two blocks does harm, and each alternating rule is a roundabout way of disabling it. Across the 19 models the link between "relies on the layer" and "final loss" is strong (correlation 0.90).
5. The gain is in learning, not in generalising. The better models also have a lower loss on the training text, by a wide margin:

   model	loss on training text	loss on held-out text
   plain matrix product	3.150	3.482
   standard transformer	3.012	3.350
   offset (x+2, y+1)	2.858	3.258
   staircase through cell, three states	2.856	3.262
   feed-forward removed from blocks 1–2	2.785	3.220

   A regulariser would raise the training loss and lower the held-out loss. Here both fall together, so the alternating models simply learn faster. (In 1,000 iterations a model sees less than half of the training text, so "training text" here is mostly unseen too; the held-out text comes from different documents.)
6. The models became tolerant of shifts they never saw. Shifting the result of blocks 1–2 by any offset costs the plain model 0.67 to 0.90. A model trained on two states loses about 0.10 on shifts it never saw, and the four-state model only 0.02 to 0.03:

   model	a shift it was trained on	shifts it never saw
   plain product (trained on none)	–	+0.67 to +0.90
   offset (x+2, y+1), two states	+0.003	+0.09 to +0.10
   offset (x+1, y+1), two states	−0.001	+0.05 to +0.16
   offset, three states	+0.004	+0.02 to +0.10
   offset, four states	+0.002 to +0.006	+0.02 to +0.03

   This is the same finding seen from another side: a model that hardly uses the layer does not care where its results land.
7. It also explains the queens mask. There the mask changed on every iteration in all six blocks, and the models learned to ignore every feed-forward: switching off blocks 1–2 changes nothing, and the outputs of the later layers are four to ten times smaller than in the other models. But blocks 3–6 are layers the model needs (switching them off costs 1.0 to 1.6 in the other models). Disabling the first two helps; disabling all six is why the queens models are so far behind.

What this means for the ideas above.

• The rotating offset and the staircase are not better multiplications. They are ways of making one layer unreliable, and the model is better off without that layer. That is why the exact rule mattered so little, and why every alternating variant landed in the same band.
• The patterns fit. A larger shift or a steeper tilt changes the output more between states, which pushes the model further from the layer. Paths through the output cell and extra states do the same. The three-state staircase through the cell, the most switched-off variant (0.002), is the best staircase.
• An exception that matters. With the local product (below), offset models also end up ignoring the layer, yet stay well behind the model without it. Being switched off by the end is not the whole explanation; the layer also costs something along the way.
• What is not known. This is one seed at 1,000 iterations, in this pyramid model, with this matrix-product layer. It is not known whether the layers stay harmful in a longer run, whether the triangle models would also gain from dropping their first two feed-forward layers, or whether a different learning rate would make the layers useful. The obvious next run is the removed-layer model at 4,000 steps with 3 seeds, next to the transformer.

Letting the model decide: a gain that starts at zero

Instead of removing the feed-forward of blocks 1–2, or disturbing it with an offset, the layer's output is multiplied by one learnable number per layer, a gain, that starts at zero. The layer begins switched off, and training turns it up only as far as it helps. This is the ReZero idea from the published work below. Nine runs, seed 0, 1,000 iterations, same text. The staircase is the two-state one, ±26.5° in turn, with its paths through the output cell. The triangle is the wrapped triangle layer of section 4, sweeping all three directions on every iteration (3 of 3) or two of the three in rotation (2 of 3).

• Given the choice, the model keeps the layer almost entirely off. No gain grew beyond 0.11, and most stayed below 0.05. This agrees with the removal result, reached a different way: the model was free to use the layer and declined.
• The queens-mask layer is not used at all. Its gains stay at about 0.001 to 0.005 in both models, and switching the layer off in the trained pyramid changes the loss by 0.000. A layer whose mask changes at random every iteration has nothing dependable to offer, so nothing is asked of it.
• In the standard transformer this is no improvement: 3.417 with the queens mask, 3.408 with the staircase. With gains this small the model is in effect the transformer without its first two feed-forward layers (3.378), and it ends 0.06 to 0.07 behind the full transformer (0.035 with the triangle). The ordinary feed-forward layers that were replaced did some good.
• The alternating staircase is barely used either. In the pyramid its gains settle at about 0.01, its output is a twentieth of its input, and switching it off afterwards costs 0.013. In the transformer its gains reach about 0.05, and the result is still worse than the plain transformer.
• In the pyramid, what sits in blocks 1–2 hardly matters once it has a gain. Full product 3.192, staircase 3.193, queens mask 3.205, triangle 3.212, nothing at all 3.220. These span 0.03, which is the noise.
• With a gain, "2 of 3" is as good as "3 of 3". The triangle scores 3.212 either way in the pyramid, and 3.385 against 3.389 in the transformer. Earlier, with the triangle at full strength in all six blocks, skipping a sweep cost 0.10 to 0.13. The difference is the gain: a layer running at a few percent of its strength cannot do much damage, whatever it computes.
• The model trusts the incomplete triangle less. The gains for 2 of 3 are smaller than for 3 of 3 (0.02 against 0.04 to 0.05 in the pyramid, 0.09 against 0.11 in the transformer), and switching the layer off afterwards costs 0.018 against 0.084.
• The triangle is the layer the transformer uses most, and it still does not beat the plain transformer. Its gains reach 0.11, the largest of the nine runs, and 3.385 is the best of the transformer variants here. That is 0.035 behind the full transformer, which is inside the noise, with 20% fewer parameters (the triangle of side 64 is much smaller than the feed-forward layer it replaces).
• The full matrix product is used at about 3% strength, and that is the best model so far: 3.192. Its gains settle near 0.03. Switching the layer off afterwards costs 0.095, so the small dose is doing something. It is 0.029 better than removing the layer and 0.159 better than the transformer.
• How much of that is noise. Two of these runs give a direct reading. The pyramid with an unused queens layer is in effect the removed-layer model with the same starting weights, and they differ by 0.015 (3.205 against 3.220). The transformer with an unused queens layer is in effect the transformer without two feed-forward layers, with different starting weights, and they differ by 0.04. So 0.029 is not a reliable difference, and "a small dose beats none" is a lead, not a result.
• What the gain alone does not settle is whether the layer is unwanted because it multiplies or because it squeezes 938 features into 100 values. A wider matrix was tested for that in “Where the matrix product helps, and where it hurts” below: it is unwanted too.

A local product: only the nearby cells

The best model so far has no feed-forward half in blocks 1–2 (their attention half stays) and plain matrix products in blocks 3–6. In a matrix product, every output cell multiplies its whole row with its whole column. Here the cell only uses the cells next to it: about N/3 of them along its row and N/3 along its column, centred on the cell and wrapping around at the edges. That makes it similar to a small cross-shaped convolution.

h = half the window          # window = 2h + 1 ≈ N/3 cells: 3 for N = 10 and 11, 5 for N = 13
dest_cell[x, y] = sum over d in -h..h of
    matrix[y][(x + d) % width]  *  matrix[(y + d) % height][x]

Six runs, each the local version of a model above: same seed, same starting weights, same text, 1,000 iterations. With the full product, an offset can be read two ways that come to the same thing. With the local product they differ, so both were run: result shifted moves the whole cross to another cell; arms moved follows the original rule literally, taking the row arm from row y + yoffset and the column arm from column x + xoffset while the output cell stays put (the third grid above).

• The local product works about as well as the full one. In the best model it ends at 3.255, which is 0.035 behind the full product and still 0.095 ahead of the transformer. With products in all six blocks and no offset it is 0.022 ahead of the full product. One difference each way, both small: on one seed they cannot be told apart.
• So here fewer multiplications are nearly free, unlike the queens mask. The local product does about a third of the multiplications (300 instead of 1,000 for a 10 × 10 matrix) and a step is about 18% faster. The difference from the queens mask: the local window is the same on every iteration and keeps a result for every cell. The queens mask changed at random and left most cells empty.
• The offset trick is much weaker on the local product. With the full product, an offset improved the model by about 0.2 and put it ahead of the transformer. With the local product the best offset version gains 0.08 (3.461 → 3.385), one version makes it worse (3.495), and none reaches the transformer (3.350).
• Four states beat two, and moving the arms beats shifting the result, but the differences between the offset versions (0.01 to 0.11) are within what one seed can show.
• These models did switch the layer off, and it was not enough. In the three better offset versions, switching off the feed-forward of blocks 1–2 afterwards costs only 0.01 to 0.04, as low as in the full-product models. Yet they end 0.13 to 0.17 behind the model that never had the layer (3.255). So "the model ends up ignoring the layer" is not the same as "the layer was never there". The layer still costs something while the model is learning to ignore it, and with the local product that cost was not recovered in 1,000 iterations.
• This weakens the offset and strengthens removal. Rearranging a layer's output is an indirect and unreliable way of getting rid of it. Removing the layer worked with both the full and the local product.
• With the layer removed, the local model leans harder on what is left. Switching off the feed-forward of blocks 3–6 costs 2.5 bits/char in the local model, against 1.6 in the full one.
• One seed at 1,000 iterations. The local product is a candidate for a cheaper layer, not a result. A window of N/3 was the only size tried.

Few bits: how little precision do these models need?

A model stored and run with very few bits per number is smaller and can run without floating-point hardware. Three measurements on the 1,000-iteration models above, one seed each. The matrix-product feed-forward has a head start: it only multiplies and adds, so on whole numbers it is exact. A standard feed-forward layer needs an activation function (GELU), which is a floating-point function.

1. Rounding the weights after training. Every weight matrix of the trained model is rounded to b bits, with no retraining.

2. Training with few-bit weights. Models that really run at one or two bits are trained for it: the weights are rounded in every forward pass during training, as in BitNet. "Three values" means every weight is −1, 0 or +1 times one scale per matrix; "one bit" means ±1.

3. Rounding the numbers that flow through the model. The input to every attention, feed-forward and output layer is rounded to b bits at evaluation. Every model behaves the same, including the few-bit-weight ones:

• The matrix-product models tolerate few bits about twice as well as the transformer. Rounded to 3 bits after training they lose 0.08 against 0.14. Trained with one-bit or three-valued weights they lose 0.09 to 0.14 against 0.24 to 0.26.
• With one-bit weights the matrix-product model matches the full-precision transformer: 3.361 against 3.350. Its 1.87M weights then fit in 0.23 MB, where the transformer's 2.25M weights take 9 MB as 32-bit floats.
• One bit is as good as three values. In both models the two are within 0.02 of each other.
• Down to 5 bits, rounding costs almost nothing, for weights and for the values flowing through the model alike (0.01 or less). At 4 bits it costs 0.02 to 0.03. Below that it has to be trained for.
• How close this is to a model with no floats. With three-valued weights and 5-bit inputs, every weight multiplication is a small whole number times −1, 0 or +1, and the matrix-product feed-forward is exact whole-number arithmetic. What still needs floats: the normalisation layers (a mean, a variance and a square root), the softmax in attention (an exponential) and one scale constant per weight matrix. An integer-only forward pass has not been built; the numbers above simulate few-bit values inside float arithmetic.
• Not tested: few-bit weights and few-bit values trained together, more than one seed, and longer runs. In section 4 the same question was asked of the tiny models, where the three-way weights of the triangle and cube also tolerated rounding better than an MLP's matrices.

Robustness to disruptions

A hypothesis: the triangle multiplies three things at a time (a weight and two values), so it may be more robust to disruptions than an ordinary layer. Tested on the trained 1,000-iteration models, with no retraining, by disturbing them in five ways and measuring how much the loss rises:

• noise on every weight: each weight multiplied by 1 + 0.2·ε, with ε random (mean of three draws);
• noise on the values flowing into every layer: half their typical spread added at random;
• dropped values: a random 20% of those values set to zero;
• rounding every weight to 3 or 2 bits.

• The triangle is more robust than the transformer, at the same loss. The no-wrap triangle and the transformer both score 3.35, which makes this a fair pair. The triangle loses less under every disruption: 10 to 12% less from noisy or dropped values, a quarter less from noisy weights, and about half as much from 3-bit weights.
• The biggest difference is in the weights. Rounding to 3 bits costs the transformer 0.141 and every multiplying model 0.067 to 0.075. Disturbing the values instead makes a smaller difference.
• It is not because the triangle multiplies three things. The triangle's three factors are one weight and two values, so it multiplies values in pairs, exactly as the matrix product does. And the matrix-product models are as robust to rounding as the triangle. What the robust models share is a feed-forward layer that multiplies values with each other instead of passing them through an activation function.
• For noisy or dropped values, the triangle has an edge of its own. The matrix-product model without its first two layers is about as fragile there as the transformer, while the no-wrap triangle is 10 to 12% better.
• A weaker model has less to lose. The matrix-product model with a product in every block looks the most robust of all, but its loss is 0.13 higher to start with, and a less confident model loses less when disturbed. That is why the comparison at equal loss, triangle against transformer, is the one to trust.
• One seed, and disruptions applied only after training. Training with the disruptions, as the few-bit section did with one-bit weights, is a different test.

All 1,000-step models side by side

Every full model on this page that was trained for 1,000 iterations (55 of them), shown as its difference from the standard transformer. Bars to the left are better than the transformer, bars to the right are worse.

The runs come from three batches trained on slightly different versions of the text, so the raw losses cannot be compared across batches. Each model is therefore compared with the transformer trained in its own batch, on the same text.

• The best models are the ones with a layer taken out or turned almost off. The six at the top are pyramid models whose blocks 1–2 layer has a learned gain that stays near zero (full matrix product 0.159 better than the transformer, alternating staircase 0.157, queens mask 0.146, the triangle 0.138 with either 3 or 2 of its sweeps) and the one with no feed-forward in blocks 1–2 at all (0.130). These six are within noise of each other. The local-product version of the last one follows at 0.095 better.
• Ten more models are ahead of the transformer. Eight are beyond the too-close-to-call band (0.059 to 0.093 better): four alternating offset rules, three staircases through the output cell (three states, ±26.5° and ±33.7°) and the ±26.5° staircase from the edge. Two sit at the band's edge (0.045 and 0.047 better). All ten are matrix products that alternate between two or more states, which works by making the model ignore that same layer. Each rests on a single run.
• A middle group cannot be told apart from the transformer: the reversed triangles, the plain triangle, the ±18.4° staircases, the fixed +26.5° staircase through the output cell, the three-state staircase from the edge, the local product with a four-state offset (arms moved) and the transformer without its first two feed-forward layers all sit within 0.04 of it.
• Everything else is behind: the other fixed staircases, the plain matrix product and the other local products that keep the blocks 1–2 layer (0.05 to 0.14 worse), the centre-lowered and scaled triangles (0.08 to 0.17), the hexagonal feed-forward (0.20) and the queens masks (0.17 and 0.21 with a fixed placement, 0.40 and 0.56 with a random one).
• A warning from the wrapped triangle. Here it is 0.06 worse than the transformer and behind the plain triangle. Trained for 4,000 steps it became the best triangle, 0.013 from the transformer. Short runs can mislead.
• Read it as a map of leads, not a ranking. The models on the left are the ones worth training longer and with more seeds.

Where the matrix product helps, and where it hurts

Two questions were still open: which part of the pyramid model gives it its lead over the transformer, and whether the blocks 1–2 layer is unwanted because it squeezes 938 features into a 10 × 10 matrix. Three control runs answer both. Same seed, same starting weights, same text, 1,000 iterations.

• The matrix product in blocks 3–6 is the source of the lead. Replacing it with an ordinary layer of the same size costs 0.10 (3.220 → 3.321). That leaves the pyramid only 0.03 ahead of the transformer, which is noise. So of the 0.13 lead, about 0.10 comes from multiplying features in the later blocks, and the pyramid's other parts (its wide early blocks, its bottlenecks, its smaller attention) contribute little.
• At equal size, the product beats the activation function. Both layers squeeze the features into 121 to 169 values and expand them again. Used that way, multiplying the values as a matrix does better than passing them through GELU. This is a narrow layer, though. A standard transformer layer expands to four times the width, and that comparison was not made.
• The squeeze is not to blame. A 30 × 30 matrix in blocks 1–2, which squeezes nothing, is unwanted too: with a gain the model leaves it at 0.01 to 0.02, and at full strength it is worse than the small one (3.547 against 3.482), despite 3M extra parameters.
• So the same layer helps late and hurts early. In blocks 3–6 the matrix product is worth 0.10. In blocks 1–2 it costs 0.26, or 0.33 when wide. One reading: the first blocks work on features that have barely been mixed across the text yet, and products of those are noise; later blocks multiply features that already carry context. That reading is a guess.

The triangle with a local window

So far the triangle layer came in two forms. Without wrap, the line at index i holds n − i weights, so outputs collect anything from 1 to n products. With wrap, every line holds n weights. A third form keeps the wrap but shortens every line: each output collects only the weights in a window of n/2 + 1 cells centred on its own index, wrapping around at the ends. For the side-50 triangle that is 26 weights per output instead of 50.

All three were trained as the pyramid model with the triangle layer in every block: 1,000 iterations, seed 0, the same text as the standard transformer run.

• The window is the weakest of the three here: 3.431. It is 0.080 behind the transformer, 0.023 behind the wrapped triangle and 0.080 behind the triangle without wrap.
• Against wrap, halving the products cost almost nothing. The window collects half as many products per output as wrap, loses 0.023, which is inside the noise (0.015 to 0.04), and makes an iteration 17% faster. That matches the local matrix product above: a consistent local window is cheap.
• Against no wrap, the same number of products arranged evenly did worse. The window and the no-wrap triangle do the same number of products. The window gives every output 26 of them; no wrap gives some outputs 50 and others 1. The uneven one is 0.080 better at 1,000 iterations. That is the opposite of what the 4,000-step runs showed, where the evenly loaded (wrapped) triangle won.
• The likely reason is the run length. At 1,000 iterations the no-wrap triangle has led every time it was measured, and it lost that lead by 4,000. Section 5 found that outputs collecting many products learn much faster, so an uneven layer gets off to a quicker start. Whether the window overtakes it in a longer run has not been tested.
• None of the three beats the transformer. The no-wrap triangle is level with it (3.351 against 3.350) and the other two are behind. The matrix-product pyramids above (3.19 to 3.22) are 0.13 to 0.24 ahead of all three triangle pyramids.
• Unlike the matrix product, the triangle is used in the early blocks. Switching off the feed-forward of blocks 1–2 costs these models 0.24 to 0.29. The matrix-product models that did best had learned to ignore that layer. Whether a triangle pyramid would also improve without its first two layers has not been tested.
• The triangle is slow whichever way its weights are collected: 0.30 to 0.48 s per iteration on four threads, six to ten times the transformer.

Three of three, at uneven rotating strengths

Skipping one of the triangle's three sweeps per iteration made the model worse. A gentler version keeps all three sweeps on every iteration but gives them uneven strengths that rotate:

contributions = [3/6, 2/6, 1/6]
a = contributions[i % 3]            # i = iteration
b = contributions[(i + 1) % 3]
c = contributions[(i + 2) % 3]

The three are scaled by 3 so that they average 1 and the layer's output keeps its size: the sweeps run at strengths 1.5, 1.0 and 0.5, and the roles rotate every iteration. On the same scale, "2 of 3" is 1.5, 1.5, 0 and plain "3 of 3" is 1, 1, 1. Evaluation always uses all three sweeps at equal strength. The triangle layer is in every block; 1,000 iterations, seed 0, same text.

• Uneven is better than skipping, and worse than equal. With wrap it costs 0.041 against 0.102 for 2 of 3; without wrap 0.103 against 0.178. So keeping the third sweep at reduced strength recovers a good part of what skipping it lost, but not all.
• The cost grows with how uneven the strengths are. Equal, then 1.5 / 1.0 / 0.5, then 1.5 / 1.5 / 0: each step makes the model worse, in both triangles. There is no sign of a benefit from rotating the strengths, only a cost that scales with the size of the disturbance.
• A model trained on uneven strengths still prefers equal ones. Evaluated in any of its three uneven states, the wrapped model scores 3.462 to 3.466, against 3.448 with equal strengths. With one sweep removed altogether it scores 3.496.
• The no-wrap triangle is hurt more than the wrapped one (0.103 and 0.178 against 0.041 and 0.102). It is the better model at this run length with equal strengths, and it loses that advantage as soon as the strengths rotate.
• All of these are behind the standard transformer, by 0.10 with uneven strengths and 0.16 to 0.18 with 2 of 3.
• The same lesson as before. Every scheme that changes what a layer computes from one iteration to the next has cost something, unless the layer was one the model was better off without. The triangle layers are ones the model uses, so disturbing them hurts. One seed each.

Compared with the standard transformer

The best models of this section are the pyramid with matrix products in blocks 3–6 and a blocks 1–2 layer that is removed, or switched almost off by a gain ("the pyramid" below). They lead the standard transformer after 1,000 iterations. A lead per iteration is only half the story, because the two models do not take the same time per iteration.

How fast is each model? That depends on what is measured. The first version of this comparison used wall-clock time, which measures the code as much as the model, so three things are reported here: the arithmetic each model needs, which does not depend on the code; the time per iteration after optimising the code; and the loss at equal training time.

• In arithmetic the pyramid is the cheaper model. It needs 1.77M multiplications per token against 2.31M for the transformer. The matrix products themselves are 0.4% of that.
• It is still slower, because of its width. Each token carries 938, 762 and then 592 values through the pyramid, against 184 in the transformer. Everything that touches those values without being a matrix multiply (normalising them, adding them up, keeping them for the backward pass) costs 3 to 5 times more, and on this machine those operations are far slower per value than a matrix multiply.
• Optimising the code helped a little, not a lot. Three changes were made: the small matrix products are computed in one vectorised step instead of one matrix at a time, a normalisation that fed a removed layer is skipped, and the bottleneck uses a faster lookup. The pyramid went from 1.7× to 1.6× the transformer's time per iteration on the CPU. More threads did not change the ratio.
• A GPU does not rescue it. At the small batch used here the gap narrows to 1.3×, but with larger batches it widens to more than 3×. The width is a cost of the design, not only of this code.

Loss at equal iterations and at equal time. Both models were also trained for 2,000 iterations, and the pyramid for 1,250, which takes it as long as the transformer needs for 2,000. Same seed and text.

• Per iteration the pyramid is ahead; per second the transformer is. In the same 100 seconds the transformer does 2,000 iterations and the pyramid 1,250, and the transformer ends 0.12 better. Before the code was optimised that gap was 0.21.
• The pyramid's lead shrinks as training gets longer. With the layer removed it is 0.13 after 1,000 iterations and 0.034 after 2,000, which is inside the noise. With the gain it is 0.16 and then 0.07. The triangle models showed the same pattern and ended behind the transformer at 4,000 steps. Whether any lead survives a 4,000-step run has not been tested.
• None of the block 1–2 ideas is the source of the lead. Offsets, staircases, queens masks and gated products all end in the same place as having nothing in blocks 1–2 (3.19 to 3.22). The lead belongs to the rest of the pyramid model, and the control above places most of it in the matrix products of blocks 3–6: swapping them for an ordinary layer of the same size removes 0.10 of the 0.13.
• The ideas do not transfer to the standard transformer. A queens-mask, staircase or triangle layer with a gain in its blocks 1–2 made it worse or no better (3.39 to 3.42 against 3.350), because the ordinary feed-forward layers they replaced were doing some good.
• The model never turns the gain up. After 2,000 iterations the gains are still about 0.02. More training did not make the layer wanted.
• Few bits is where the matrix product has its clearest advantage. It loses half as much as the transformer when its weights are rounded or trained at one bit, and its feed-forward needs no activation function. That advantage does not depend on speed.
• The noise is measured, not guessed. Models that are identical in effect differ by 0.015 with the same starting weights and 0.04 with different ones, and merely changing the rounding order moved one run by 0.004. Any gap on this page below about 0.05 between single runs should be read as no difference.
• A narrower pyramid is the obvious next design. The lead comes from the matrix products in the later blocks, and the cost comes from the width. A model that keeps the products and drops the wide early blocks has not been tried.

Takeaways

The common thread: the rectangle is the attractor. A matrix is a rectangle because a linear map pairs every input with every output. Shapes drawn inside that rectangle drift back towards rectangles when you multiply. Every centred shape becomes its bounding box within 2–3 products. Other shapes stay themselves only when the multiplication changes. Convolution adds shapes instead of composing them, and keeps every convex polygon. And when the polygon describes how matrices are wired together (a ring), what matters is not the shape but its parity.

Surprises

1. A pentagon can be a stable matrix shape, a hexagon cannot. The more symmetric shape loses. A stable shape is a box cut by at most one diagonal line, and a rectangle cut by one line has at most 5 corners.
2. Geometry gives the exact cost. Multiplying two P-shaped arrays needs exactly 4|P| − B − 3 multiplications, by Pick's theorem. So the number of lattice points on the edge sets the cost. That's why pentagons, octagons and disks, which can't be regular on a lattice, are about 4% more expensive. The crystallographic restriction shows up as a price.
3. Odd versus even decides everything for rings. Square, hexagon and octagon rings are incompressible: every fused algorithm needs all 2^k terms. Triangle, pentagon and heptagon rings are compressible. Strassen's speed-up is, in this light, an odd-cycle phenomenon.
4. Hexagons are made of triangles: a triangular array correlated with its mirror image gives a hexagonal array.
5. 2D is special. A doubled lattice polygon never has gaps in 2D, but a doubled tetrahedron can (Reeve), so 3D shaped arrays are less tidy.
6. The natural symmetric generalisation of Strassen doesn't simply appear for the pentagon. A rotation-symmetric 31-term fit froze at error 0.0003. The only lead below 31 is an unconfirmed 30-term approximate decomposition.
7. The three-way weights tolerate coarse rounding, the projections don't. The cube's 49k three-way weights at 2 bits hurt less than one 33k MLP matrix at 2 bits. The sensitive part is the input projection, whose errors are multiplied together.
8. Without wrap-around, a hexagon's centre learns about 30× faster than its corners, because more products land there. Wrap-around makes learning uniform.
9. Hexagonal blurs are exactly round up to 4th order at every step; square blurs only become round slowly.
10. In a whole model, the triangle beats the hexagon by a wide margin. A pyramid built on hexagonal feature cells with 3 → 1 bottlenecks came close to a standard transformer once its feed-forward was the triangle layer (the wrapped triangle is 0.013 bits/char behind at 4,000 steps, within seed noise), and stayed 0.2 behind with the hexagon's local mixing. The pyramid was never the problem.
11. The "centre" means different things in the two shapes. In the triangle the long lines are signal (lowering them hurts, raising them is free); in the hexagon the centre-heavy sums were neither harmful nor needed (1/√ balancing changes nothing).
12. A centre-heavy hexagonal layer learns an importance order without being asked. In a language model, trimming its outer rings hurts 2–3× less than trimming at random. Meanwhile training partly fights the non-uniformity, growing the edge weights to make up for weaker edge activations.
13. Uniform wins in both shapes. Given the choice between a centre-heavy and a uniform (wrapped) layout, the uniform one won for the hexagonal layer and, in 4,000-step runs, for the triangle feed-forward (2.739 against 2.778 bits/char). No reweighting of the centre and edges beat it.
14. The shape's own weights are about 1% of the model, yet changing how they are wired (wrap or not) moved the loss more than any centre/edge weighting. The wiring matters more than the weights.
15. All three sweep directions are needed. Training the triangle with only two of its three sweeps per step, rotating which one is skipped, cost 0.10–0.13 bits/char and saved only about 15% of the step time. Each direction carries information the other two do not.
16. More directions are not automatically better. A wrapped square swept along rows, columns and both diagonals, at the same parameter count, ended 0.028 bits/char behind the transformer and 0.015 behind the 3-direction wrapped triangle. Dropping a direction hurts, but adding one does not help: the trained square uses all four directions equally, yet each one matters less than in the triangle (0.23–0.25 against 0.40 when switched off). The extra direction divides the work instead of adding to it.
17. The best trick turned out to be a subtraction. Rotating offsets and staircase paths made a matrix-product model beat the transformer at 1,000 steps. Opening the trained models showed why: they had learned to ignore the layer the tricks were applied to. Removing it outright did better than every trick (3.220 against 3.258, transformer 3.350; one seed). What is removed is the feed-forward half of blocks 1 and 2. Their attention half stays, so the model keeps its six blocks but has four feed-forward layers instead of six, and 17% fewer parameters. The standard transformer does not gain from the same removal (3.378 against 3.350), so the harm is specific to the matrix-product layer.
18. A model with one-bit weights matched the full-precision transformer. Trained with every weight restricted to ±1, the matrix-product model scores 3.361 against 3.350 for the transformer with 32-bit weights, and the transformer loses twice as much when it is restricted the same way (one seed, 1,000 steps).
19. Ahead per iteration, behind per second. The best pyramid model beats the standard transformer by 0.16 after 1,000 iterations and 0.07 after 2,000. It needs fewer multiplications per token, yet takes 1.6 times as long per iteration even after optimising the code, because each token carries a vector 3 to 5 times wider. In the same 100 seconds the transformer ends 0.12 better.
20. Exact search hits a wall quickly. A SAT solver settles the triangle instantly. But it couldn't prove within 15 minutes that the square ring needs more than 15 products, a fact that a one-line flattening argument proves.

The language-model experiments: what stood out

Section 6 grew long. These are its main results, most surprising first. Each says how far it can be trusted: solid means 4,000 steps with 3 seeds; one seed means a single 1,000-step run, usually paired with a control that started from the same weights.

1. The tricks worked by switching a layer off. Rotating offsets and staircase paths made a matrix-product model beat the transformer. The trained models turned out to ignore the layer the tricks were applied to, and removing that layer did better than any trick (3.220 against 3.258). One seed.
2. The same layer helps late and hurts early. A matrix product as the feed-forward layer is worth 0.10 in blocks 3–6 against an ordinary layer of the same size, and costs 0.26 in blocks 1–2. Making the early layer wider makes it worse, so it is not a matter of size. One seed.
3. Asked directly, the model says no. With a learnable gain that starts at zero, the model leaves every blocks 1–2 layer nearly off: queens mask 0.005, staircase 0.01 to 0.05, matrix product 0.03, triangle 0.02 to 0.11. That held at 2,000 steps and for a wide matrix. One seed each.
4. A model with one-bit weights matched the full-precision transformer. The matrix-product model with every weight restricted to ±1 scores 3.361 against 3.350, and loses half as much as the transformer does under the same restriction. Its feed-forward needs no activation function, so on whole numbers it is exact. One seed.
5. Ahead per iteration, behind per second. The best pyramid model leads the transformer by 0.16 after 1,000 iterations and 0.07 after 2,000. It needs fewer multiplications per token (1.77M against 2.31M), but each token carries a vector 3 to 5 times wider, and that makes an iteration 1.6 times as long even after optimising the code. In the same 100 seconds the transformer ends 0.12 better. One seed.
6. A fixed tenth of the multiplications costs almost nothing; a random tenth costs everything. A local product that uses only the cells near the output cell matched the full product, and so did a queens mask kept the same on every iteration (0.036 behind with a tenth of the products). The same queens mask redrawn at random every iteration was the worst of the 1,000-step models. What matters is that the layer computes the same thing every time, not how much it computes. One seed.
7. Ending up ignoring a layer is not the same as never having it. With the local product, offset models switched the layer off and still finished 0.13 to 0.17 behind the model without it. One seed.
8. How much a disturbed layer hurts depends on how loud it is and how big the disturbance is. Skipping one of the triangle's three sweeps cost 0.10 to 0.13 at full strength in every block (solid), and nothing at all behind a gain of a few percent (one seed). Running all three at uneven rotating strengths cost about half as much as skipping one (one seed).
9. Uniform beats centred. For the triangle and the hexagon, the wrapped layout in which every output collects the same number of products came out best. No reweighting of centre against edge beat it. Solid for the triangle; the hexagon was tested only in a tiny model. At 1,000 steps the picture is the reverse: an evenly loaded triangle with a local window was 0.08 behind the uneven no-wrap triangle doing the same number of products (one seed). Uneven layers start faster; even ones caught up later.
10. More directions did not help. A square swept along rows, columns and both diagonals ended behind the transformer and no better than the three-direction triangle. The fourth direction was fully used, but it divided the work instead of adding to it. Solid.
11. Short runs mislead, and the noise is now measured. Rankings from 1,000 steps and 2 seeds were wrong twice at 4,000 steps. Models that are identical in effect differ by 0.015 to 0.04 between single runs, so a gap below about 0.05 means nothing.
12. What survived the longest test. At 4,000 steps with 3 seeds the standard transformer beat every shaped layer that was run at that length. The best models of the later experiments have not been run that long.

Where this is useful

• Attention masks are shapes. A causal mask is a triangle, which is closed, so stacking causal layers stays causal. A sliding-window causal mask is a band cut by the triangle. It is not closed: each product widens the band. That widening is exactly how a stack of sliding-window layers gets a receptive field of about (layers × window). Pick these two masks in section 1 to see it.
• Storage formats that stay closed: triangles, trapezoids and the triangle-in-a-window pentagon can be multiplied without the result leaving the format. Bands and centred shapes need re-allocation after every product.
• Choosing an array shape for convolution data: hexagons for round, isotropic data (images, sensors, game maps; 13.4% fewer samples); triangles for polynomials of bounded total degree. The cost formula 4|P| − B − 3 lets you budget the work exactly.
• Mixed-precision storage of shaped layers: keep the small input projection precise and store the three-way weights at 3 bits. In the tiny model the cube then beat the gated block at the same storage (3.530 vs 3.602 bits/char at about 200k bits).
• Hexagonal data (geospatial H3 cells, game maps, detector arrays): a ring-1 + skip + random sparse mask connects a radius-8 map in 5 layers at 9.5 links per cell, against 16 layers for neighbours only. Wrap-around or not is a design choice: uniform learning, or a centre-first curriculum.
• Elastic width for free-ish: a non-wrapped hexagonal feed-forward layer orders its own outputs by importance, so outer rings can be trimmed at inference when compute is short, at a fraction of the cost of trimming at random.
• Testing a new layer shape: 1,000-step runs with 2 seeds ranked these variants wrongly twice (seed spread up to 0.19). At 4,000 steps with 3 seeds the spread fell to at most 0.045 and the rankings held. Budget for that before trusting a gap of a few hundredths.
• Small, float-light language models: a feed-forward layer built from a matrix product needs no activation function, so with whole-number inputs and −1/0/+1 weights it is exact integer arithmetic, and these models lost half as much as the transformer when trained with one-bit weights. Normalisation and softmax would still need integer replacements.
• Tensor networks with loops: an even loop can't be shortcut by any fused algorithm, so the order of pairwise contractions is all that matters. Odd loops leave room for Strassen-like tricks.
• Shaped weights in a language model (triangle, cube; the tetrahedron was too slow to judge) train fine and beat a plain MLP, but not SwiGLU. The cube did best. Their selling point would be 6 multiplications per weight read for memory-bound inference, which is untested without a GPU kernel. The triangles that already pay off in large models live over tokens (2-simplicial attention) or residue pairs (AlphaFold).

How new is this?

Most of it is known in some field: triangular matrices, receptive fields, Newton polytopes and Minkowski sums, Pick's theorem, hexagonal sampling, and Christandl–Zuiddam's cycle tensors. What this page adds is the side-by-side view, the classification of stable matrix shapes (at most 5 corners), the closed cost formula 4|P| − B − 3 for shaped convolution, and the unconfirmed hint that the pentagon ring might be approximated with 30 terms. Also new are the small experiments: shaped weight layers swept three or four ways; the finding that their three-way weights are robust to low precision while the input projection is not; the hexagonal wrap / no-wrap learning-speed contrast; the emergent importance order of a centre-heavy hexagonal layer; and a hexagonal pyramid language model whose wrapped-triangle feed-forward lands within seed noise of a parameter-matched transformer. The isotropy of hexagonal blurs and the power of random long-range links (BigBird, small-world graphs) are known.

How the language-model findings compare with published work

On raw performance there is no comparison. Published transformers reach just under 1 bit per character on the standard enwik8 benchmark (0.97 for the Compressive Transformer with 277M parameters, 0.98 for the Hourglass model). The models in section 6 have about 2M parameters, train for 1,000 to 4,000 steps on 2.5 MB of text and score 2.7 to 3.5. Nothing here competes with the state of the art. The findings are about how small models behave relative to each other.

What is less standard. These four do not appear in the papers above. A wider literature search was not done, and each rests on one seed and 1,000 iterations.

• A clean case of an improvement that was really a removal. The offsets and staircases looked like better multiplications. Switching the layer off in the trained models, and then removing it, showed they worked only by disabling it. Running that control is ordinary good practice; having so clear-cut an example is less common.
• Ending up ignoring a layer is not the same as never having it. With the local product, the offset models switched the layer off and still ended 0.13 to 0.17 behind the model without it.
• The same layer helps late and hurts early. Removing the matrix product from blocks 1–2 helps by 0.26, and the same removal costs the transformer 0.03. In blocks 3–6 the matrix product beats an ordinary layer of the same size by 0.10. Published comparisons of feed-forward variants usually swap every layer at once, so a dependence on position would not show.
• A one-bit matrix-product model matched a full-precision transformer at the same parameter count and number of steps (3.361 against 3.350), and lost half as much as the transformer did under the same restriction.

A mundane explanation, tested and ruled out. In blocks 1–2 the matrix-product layer squeezes 938 features into a 10 × 10 matrix, which is 100 values, about a ninth of the width. The harm could have come from that squeeze. It does not: a 30 × 30 matrix, which squeezes nothing, is just as unwanted and does more harm at full strength (section 6, “Where the matrix product helps, and where it hurts”).

What current practice would suggest trying. Give the layer a learnable gain that starts at zero, so the model switches it on only if it helps. That is ReZero (Bachlechner et al. 2020); LayerScale is the per-channel version (Touvron et al. 2021). It is the principled form of what the offsets did by accident. It was run in “Letting the model decide” in section 6: the gains stay near zero (about 0.03 for the full matrix product, 0.001 to 0.005 for the queens mask), so the model keeps the layer almost off.

Pros and cons by shape

So can square matrices be replaced by n-gons?

• As general linear maps, no. A linear map pairs every input with every output, and that table of pairs is a rectangle. Any other shape is a restriction of it. Only restrictions with at most 5 corners stay closed under multiplication, and the useful one is the triangle, which numerical linear algebra already exploits everywhere.
• As data with their own multiplication, yes. For convolution-type products (polynomials, filters, images, hex grids), triangles and hexagons are as natural as squares: the product keeps the shape, and the cost is exactly 4|P| − B − 3. Hexagons are the better choice for round, isotropic data; triangles for polynomials of bounded total degree.
• As the multiplication pattern, odd polygons are where the shortcuts live. Every even ring is incompressible. Every odd ring inherits a speed-up from Strassen's triangle, and the pentagon's exact cost is still open.

What is proven, what is numerical