Your count() returns the number of parent records, while your query uses limit() with limits the result of the entire query, which due to the JOIN contains duplicate parent records. Which means in case of a many relation, you may end up with less parent records after hydration.
From an ORM point of view, it is not logical to want to use pagination in combination with relations, as you will get a result like: A - 1
A - 2
B - 1
B - 2
B - 3
so you're not paginating posts, your paginating post categories. This example also shows why you can get 2 parent records when per page = 5, as the the first parent yields two records, the second three, making a total of five.
You can use rows_offset() and rows_limit() instead of offset() and limit() to force the limits on the parent only, which I think is your intended behaviour, 5 parent records, each one will all their categories.