The mutations counter is a 32 bit signed integer. If your pet has 1 mutation, in binary he has 00000001 mutations. If you mate them with another 1 mutation pet, he will have 2, or 00000010 mutations.
If you inbreed that generation, you have:
00000100 mutations, or 4.
Do it again, you have 00001000 mutations, or 8.
As you can see, every doubling of mutations moves the 1 left.
Now, for a little bit of computer fun, computers don't use the 32nd bit of a signed integer as a quantity, they use it as the positive or negative sign. 0 = positive. 1 = negative.
So, what happens after our 31st generation of inbreeding?
01000000 00000000 00000000 00000000 (1,0731741,824 mutations) +
01000000 00000000 00000000 00000000 (1,0731741,824 mutations) =
10000000 00000000 00000000 00000000 ( -2,147,483,648 mutations)
Yes, that is negative 2,147,483,648 mutations. (for computer math reference, negative 1 is binary all 1s, all 0s is the 'biggest' negative number) if you inbreed that generation, the 1 is pushed off the 32 bits containing the integer and we're back at 0.