I always thought elves were supposed to be small. Dictionary.com agrees on that: Link
Here's what the OED says:
The name of a class of supernatural beings, in early Teutonic belief supposed to possess formidable magical powers, exercised variously for the benefit or the injury of mankind.They were believed to be of dwarfish form, to produce diseases of various kinds, to act as incubi and succubi, to cause nightmares, and to steal children, substituting changelings in their place. The Teutonic belief in elves is probably the main source of the mediæval superstition respecting fairies, which, however, includes elements not of Teutonic origin; in general the Romanic word denotes a being of less terrible and more playful character than the ‘elf’ as originally conceived. In mod. literature, elf is a mere synonym of fairy n. and adj., which has to a great extent superseded it even in dialects. Originally elf was masculine, elven n. feminine; but in 13th and 14th c. the two seem to have been used indifferently of both sexes. In mod. use elf chiefly, though not always, denotes a male fairy.