Our Pikafish jieqi bot misjudged its reveals

The bot called a move winning with a mate in its own line

On October 2 we had game 108 of the AB-JChess match open, to show how Pikafish judged a reveal. Pikafish, as Black, turned over the piece on d10 and moved it to e9, and rated the position 18.7 pawns in its favour. Its own best line had Red mating. A correct search cannot find a mate in its own best line and still call the position winning.

Only an advisor on e9 stops the chariot's mate on f10, and an advisor was 1 of the 11 pieces it could have been. It was a chariot, and Black was mated next move. We took the examples out of that post the same day. This move led us to bug one, and looking further turned up bug two.

Bug one: for Black, the average took the best piece

When the bot considers a move that turns a piece over, it plays the move once for each piece the face-down one could be. It scores each, then averages the scores, weighted by how many of each piece are left. A reveal that wins if the piece is a chariot and gets mated if it is a soldier should score somewhere in between. Two bugs broke that average.

Before averaging, the code turns every score to one player's point of view. It picked the player from a setting that was never filled in, so it always assumed Red. For Red's reveals that was right. For Black's, the scores were upside down, and a rule meant to take the worst outcome, when the outcomes spread far apart, took the best one.

The flip is in misc.h, lines 360 to 366, set from search.cpp, line 1310 by asking whether the side to move is the "first side", a field nothing ever assigns. Game 108 was a Black reveal, so its scores went through the flip.

Bug two: cut-short scores were averaged as exact

We printed the score of every piece the bot could turn over at a reveal. In one match game, three of the four were mated:

An engine saves time by giving up on a move once it knows the move is worse than the best one so far. It then returns a bound, which means "this much, or worse". That is safe when the next step is a comparison. The bot fed these bounds into the average as if they were exact scores, and rated a move that gets mated for three of its four pieces as winning. Each piece is searched inside the window of the move above it, at search.cpp, line 1462.

Bug two is independent of bug one: it hits both colours, and AB-JChess has it without bug one.

Bug two alone hid a mate in the first game of the AB-JChess post. The bot, as Red, turned over the piece on b3 and moved it to d3. If that piece is a soldier or an elephant, Black's cannon mates at once on h1.

Weighted honestly, Black is better. The fixed bot sees the mate and plays the advisor from e2 to f3, the move AB-JChess recommends.

The fix drops the flip and searches a cut-short piece again

Both are in our copy of Pikafish's jieqi branch. Simplified, the old code scored a reveal like this:

# Before (simplified)
for piece in pieces_it_could_be:
    # bug two: cut short, this returns the window's edge
    s = -search(position_with(piece), window)
    # bug one: set from a field nobody fills in
    if mover_is_black:
        s = -s
    scores.add(s, weight = count[piece])
v = weighted_average(scores)
# for Black, "min" was really the best
if v - min(scores) > MAX_SPREAD:
    v = min(scores)

# After
for piece in pieces_it_could_be:
    s = -search(position_with(piece), window)
    # bug two fixed: only a bound, search again
    if at_root and s <= window.low:
        s = -search(position_with(piece), no_lower_limit)
    scores.add(s, weight = count[piece])
v = weighted_average(scores)
if v - min(scores) > MAX_SPREAD:
    v = min(scores)

Deleting the colour flip fixes bug one. Searching again any piece whose score came back as a bound fixes bug two. Both are in one commit, 28 lines.

The fixed bot rates reveals close to their worth, and plays about as well

We searched the 71 moves that decided Pikafish's losses in the AB-JChess match again with both bots, then played the fixed bot against the old one.

The fixed bot rates reveals close to their worth, and its strength is unchanged within the noise, perhaps slightly weaker: both intervals include zero. The fix for bug two covers only the move the bot is choosing. Deeper in its search the old averaging is still there, because re-searching everywhere cost about 150 Elo.

AB-JChess has bug two, and it rarely matters

AB-JChess is built from Pikafish and averages reveals the same way, so it has bug two. Of the 60 reveals that decided its losses to our bot, it overrated 2.

Why a September test of this averaging doesn't stand

In September we tested replacing this averaging with a separate search for each piece, and kept the averaging. That test ran with bug one in place, so for Black the worst-case fallback it credited was taking the best case, and its verdict doesn't stand.

The idea came from an outside review of the engine’s source, iwestlin/jieqi-ai. In July the averaging had chosen a reveal with a true 64% chance of winning over a safe move at 97%. Each test played the changed engine against the shipped one, colours alternating, with a referee dealing the hidden pieces so neither engine saw one.

# Shipped: inside the search, each time a move turns over a piece
v = weighted_average(s[piece], weight = count[piece])
if v - min(s) > MAX_SPREAD:
    v = min(s)              # fall back to the worst case

# Replacement: at the root, when the top move is a reveal
for move in top_four_moves:
    v[move] = average(win[move][piece] for piece in pieces_it_could_be)
play(argmax(v))

The separate searches lost clearly at 2.2 times the time, and at 7 times they still scored below even. Since then AB-JChess has beaten this engine 248-136-16 over 400 games and is now the top jieqi bot.

The last two rows are a separate bug, unrelated to the averaging: the same review found that one colour’s face-down pieces were never counted in the evaluation. We fixed it then and retuned the 43 terms it touches over 10,000 SPSA game pairs. Neither change left its margin: at 200 games ±0.07 is roughly ±50 Elo.