"Science fiction is what scientists would do if they could - if they had enough grant money, enough time, and enough brains to do the wonderful things they would like to do."
- Greg Bear

As far as I know, the first reference to a pocket-sized telephone occurs in John Jones's Dollar by Harry Stephen Keeler, published in The Black Cat magazine in 1915; see the pocket wireless phone.

The British cartoonist W. K. Haselden described a "Pocket Telephone" and the cartoon was first published in The Mirror on March 5, 1919:

(The Pocket Telephone - When Will It Ring?)

See also these references; the telephonoscope from The Coming Race (1929) by JD Bernal and Heinlein's 1948 novel Space Cadet - the portable telephone.

Famously, Nikola Tesla was quoted in an interview in 1926 in Collier's Magazine:

"When wireless is perfectly applied the whole earth will be converted into a huge brain, which in fact it is, all things being particles of a real and rhythmic whole. We shall be able to communicate with one another instantly, irrespective of distance. Not only this, but through television and telephony we shall see and hear one another as perfectly as though we were face to face, despite intervening distances of thousands of miles; and the instruments through which we shall be able to do his will be amazingly simple compared with our present telephone. A man will be able to carry one in his vest pocket."

Just to put some historical perspective on this one, in 1964 the closest thing to a portable telelphone was a radiotelephone, which was typically the size of a couple of shoeboxes.

Bell System engineers made the first transatlantic radiotelephone call in 1915; it used huge towers and lots of power. The original VHF Squad Radios from WWII were originally designed to be used while riding a horse! The first cellular phones were demonstrated in 1973.

Present day radiotelephones may operate at any frequency where they are licensed to do so, though typically they are used in the various bands between 60 MHz and 900 MHz.

For the WWII generation, a telephone was a fixed object affixed to a wall, or wired into one. It took real effort of the imagination to see a phone you could just put in your pocket.

See also the first reference to the idea of texting, the hand telegraph from Anno Domini 2000; or, Woman's Destiny (1889) by Sir Julius Vogel.