Implement Multi-Head Attention
Reported by candidates from ByteDance's online assessment. Pattern, common pitfall, and the honest play if you blank under the timer.
ByteDance reported this one in October 2026, and it's not a typical array puzzle. You're asked to hand-code multi-head attention with no projections, no masks, no backward pass. Just split rows into head slices, score, softmax, and mix the values. The output is a string per query row with six decimals. It's tagged as a design-style problem, but really it's careful simulation of a formula. If you've seen the transformer diagram, you know the shape. The risk is small details: the scale factor, the stable softmax, the formatting. StealthCoder sits as a safety net on the live OA if your mind goes blank on the indexing.
The problem
A Google AI/ML interview report asks the candidate to hand-code multi-head attention. For this exercise, implement a projection-free scaled dot-product attention forward pass. query, key and value are row-major integer matrices with the same feature width d; key and value have the same number of rows. heads divides d. Split each row into contiguous head slices. For each query row and head, compute dot products against every key slice, divide by sqrt(d / heads), apply a numerically stable softmax across the keys, and use those probabilities to combine the matching value slices. Concatenate the head outputs in head order. Return one comma-separated string per query row, with each coordinate formatted to exactly six decimal places. There are no learned projections, masks, biases, dropout or backward pass in this judged practice. Function multiHeadAttention(query: int[][], key: int[][], value: int[][], heads: int) → String[] Examples Example 1 query = [[1]] key = [[0],[1]] value = [[10],[20]] heads = 1 return = ["17.310586"] Softmax over scores 0 and 1 weights the second value more heavily. Example 2 query = [[1,0]] key = [[1,0],[0,1]] value = [[4,8],[6,2]] heads = 2 return = ["4.537883,5.000000"] Each one-dimensional head computes its own attention distribution. Constraints 1 <= query rows, key rows, d <= 40. Every row has width d; key and value have the same number of rows. 1 <= heads <= d, and heads divides d. Every entry is an integer in [-100, 100].
Reported by candidates. Source: FastPrep
Pattern and pitfall
The trick is that there's no trick, only discipline. Let dk = d / heads. For each query row and each head, take the slice [h*dk, (h+1)*dk). Dot it with the same slice of every key row, divide by sqrt(dk), not sqrt(d). That's the classic pitfall. Then subtract the max score before exponentiating so the softmax is stable. Weight the matching value slices by those probabilities and write the result into the same head position in the output row. Concatenation in head order falls out of the indexing for free. Sizes are at most 40, so the triple loop is trivially fast. The other pitfall is formatting: exactly six decimals, comma-separated with no spaces, and watch out for negative zero printing as -0.000000. If the loops tangle in the live OA, StealthCoder is the hedge that gets you unstuck.
StealthCoder is the hedge for the one pattern you didn't drill. It runs invisibly during the screen share.
You can drill Implement Multi-Head Attention cold, or you can hedge it. StealthCoder runs invisibly during screen share and surfaces a working solution in under 2 seconds. The proctor sees the IDE. They don't see what's behind it. If you're reading this with an OA window open, you're who this was built for.
Get StealthCoderRelated leaked OAs
You've seen the question.
Make sure you actually pass ByteDance's OA.
ByteDance reuses patterns across OAs. If you're reading this with an OA window open, you're who this was built for. Works on HackerRank, CodeSignal, CoderPad, and Karat.
Implement Multi-Head Attention FAQ
What's the scale factor in this ByteDance attention problem?+
Divide each dot product by sqrt(d / heads), the width of one head slice, not sqrt(d). Using the full width is the most common mistake. Example 2 has d=2 and heads=2, so each slice is width 1 and the divisor is 1.
Do I really need a numerically stable softmax?+
Yes, it's required by the statement. Subtract the max score in each row-head before calling exp. With entries up to 100 and d up to 40, raw scores can get large enough to overflow or lose precision, and the stable version gives identical probabilities.
How do I split the heads without extra arrays?+
Compute dk = d / heads, then loop h from 0 to heads-1 and use offsets h*dk through h*dk+dk-1 on query, key and value rows. Write the head's output into the same offsets of the result row. That gives head-order concatenation automatically.
How should I format the output strings?+
Format each coordinate to exactly six decimals and join with commas, no spaces. Return one string per query row. Watch for -0.000000 appearing from tiny negative values, and normalize it if your language prints it that way.
How do I prepare for this in 48 hours?+
Write it once from scratch on paper, then check it against both examples. Example 1 should give 17.310586. Focus on three things: slice offsets, the sqrt(dk) scale, and max-subtracted softmax. Nothing else in the problem is hard, so that's the whole prep.