I've been doing some thinking about how to square realism and playability without breaking the game too badly, partly inspired by Bret Devereaux's recent post on ancient and medieval heavy infantry spacing, partly by this gallery of HEMA fighters in 5-foot squares, and partly by some of my frustrations with both oldschool and 3.x positioning and their implications in different-size rooms. The post title gives away the short version of the answer - one foot squares, with man-size characters occupying a depth of two feet and a width of two or three feet (depending on armament).
From there, weapon reach becomes important - the front lines are no longer necessarily immediately adjacent, since a spear is going to let a combatant reach 6 squares beyond the space they're occupying. On the other hand, it might not let them effectively attack someone immediately adjacent, making daggers advantageous in wrestling range. The issue then is that the spearman wants to keep the dagger wielder at bay, while the dagger wielder wants to close through the spearman's effective range. The obvious solution here is to borrow attacks of opportunity from 3.x - the spearman gets to make an attack as the dagger wielder tries to pass through their range, and on a successful attack, they not only do damage but also prevent the dagger wielder from closing. Adding the option for the swordsman to retreat would also make the system somewhat more realistic, while giving a major advantage to longer weapon. So far so good, at least if we're looking at a duel on a fencing strip.
