Multi-Head Attention Forward Pass
Reported by candidates from Meta's online assessment. Pattern, common pitfall, and the honest play if you blank under the timer.
The mistake that sinks a first attempt on this Meta question, reported September 2026, is scaling by sqrt(d) instead of sqrt(d / heads). Everything else is plain loops. The task is a projection-free multi-head scaled dot-product attention forward pass over small integer matrices, with every output coordinate printed to six decimals. There's also an interview discussion tail on complexity, FlashAttention and linear attention that the function never judges. If your OA is in a day or two, this is a careful-implementation problem, not a clever-algorithm one. StealthCoder is the safety net on the live OA if you blank on the indexing.
The problem
Implement a projection-free multi-head scaled dot-product attention forward pass. query, key, and value are row-major integer matrices with the same feature width d; key and value have the same number of rows. heads divides d. Split each row into contiguous head slices. For each query row and head, compute dot products against every key slice, divide by sqrt(d / heads), apply a numerically stable softmax, and use the weights to combine the matching value slices. Concatenate the head outputs. Return one comma-separated string per query row with every coordinate formatted to exactly six decimal places. There are no learned projections or masks. Interview discussion (not judged by the function) After implementing the forward pass, analyze its time and extra-space complexity in terms of query rows q, key/value rows k, feature width d, and head count h. Explain conceptually how FlashAttention reduces GPU memory traffic while computing exact softmax attention, and how linear attention changes the sequence-length cost by using a factorized feature-map formulation. No implementation of either alternative is required. Function multiHeadAttention(query: int[][], key: int[][], value: int[][], heads: int) → String[] Examples Example 1 query = [[1]] key = [[0],[1]] value = [[10],[20]] heads = 1 return = ["17.310586"] Softmax over scores 0 and 1 weights the second value more heavily. Example 2 query = [[1,0]] key = [[1,0],[0,1]] value = [[4,8],[6,2]] heads = 2 return = ["4.537883,5.000000"] Each one-dimensional head computes its own attention distribution. Constraints 1 <= query rows, key rows, d <= 40. All rows have width d; 1 <= heads <= d and divides d. Entries are between -100 and 100.
Reported by candidates. Source: FastPrep
Pattern and pitfall
The pattern is array simulation with a numerically stable softmax. For each query row and each head, slice the contiguous d/h columns, dot against every key slice, divide by sqrt(d/h), subtract the row max, exponentiate, normalize, then take the weighted sum of the matching value slices. Concatenate heads in order. Pitfalls: using sqrt(d), skipping the max subtraction, slicing heads by stride instead of contiguous blocks, and formatting. Use exactly six decimals and join with commas and no spaces. Watch out for negative zero printing as -0.000000. Complexity is O(q*k*d) time and O(k) extra space per head, or O(q*d) for the output. For the discussion part, say FlashAttention tiles the computation in fast on-chip memory with an online softmax, so it never materializes the full q by k score matrix. Linear attention factorizes softmax into feature maps so cost scales linearly in sequence length. StealthCoder is there as a hedge if the slicing logic slips mid-assessment.
StealthCoder is the hedge for the one pattern you didn't drill. It runs invisibly during the screen share.
You can drill Multi-Head Attention Forward Pass cold, or you can hedge it. StealthCoder runs invisibly during screen share and surfaces a working solution in under 2 seconds. The proctor sees the IDE. They don't see what's behind it. If you're reading this with an OA window open, you're who this was built for.
Get StealthCoderRelated leaked OAs
You've seen the question.
Make sure you actually pass Meta's OA.
Meta reuses patterns across OAs. If you're reading this with an OA window open, you're who this was built for. Works on HackerRank, CodeSignal, CoderPad, and Karat.
Multi-Head Attention Forward Pass FAQ
What's the trick in the Meta multi-head attention OA?+
Scale by sqrt(d / heads), not sqrt(d). Each head works on its own contiguous slice of width d/heads, so the scaling uses the head width. Everything else is nested loops: scores, stable softmax, weighted sum of value slices, then concatenation across heads.
Do I need the max-subtraction in softmax with these small constraints?+
Do it anyway. Entries go up to 100 in magnitude, so dot products can be large and exp can overflow. Subtracting the row max keeps results identical and safe. It's one extra loop and it removes a whole class of wrong answers.
How should I format the output strings?+
Each query row becomes one string. Format every coordinate to exactly six decimals, join with commas and no spaces, heads concatenated in order. Check Example 2, which gives 4.537883,5.000000. Guard against negative zero showing up as -0.000000.
What complexity should I state for the discussion part?+
Time is O(q*k*d), since each head costs q*k*(d/h) and h heads sum to d. Extra space is O(k) for one row of scores if you process row by row, or O(q*k*h) if you store every score. State which one your code uses.
How do I prepare for this in 48 hours?+
Hand-trace Example 1 and 2 until the numbers match. Write the loop skeleton with head offsets, then add the stable softmax. Rehearse two sentences each on FlashAttention (tiling, online softmax, less memory traffic) and linear attention (feature-map factorization, linear in sequence length).