Nice to see that this forum has some life left in it!
SQ4 is probably my favorite game of the whole series, but for hardest I lend my vote to SQ2. I was stumped for months on "rub berries on self" - - so much so, that I finally had to break down and beg my parents to buy me the hintbook. I also hated the asteroid sequence, so that's yet another reason why 2 gets my vote.
I can understand the points about needing to walk every square inch of the mall in SQ4 to find out what happens next, however I guess I spent so much time simply wandering around the mall to explore every little detail canceled this part out as a difficulty factor for me.
I guess if you pirated the game, SQ6 would be next to impossible, considering you need to have the included PDF to figure out the logic with the Datacorder (but if you're going to take that route, then the copyright protection in SQ1VGA and SQ4 could easily apply as well).