The simple phrase that Iñigo wrote spurred a lot of good reflections!
I share Lah Ca's impression - the music takes precedence for me as well. But I am also attracted by the engineering behind the machines, the ingenuity of some of them that resulted or not in the improvement of reproduction. And, as mentioned before, I am most of the time more impressed by the sound of a vintage recording played on a gramophone than with modern transfers. There is the physical impact caused by the horn - especially coming from something like an EMG/Expert - that is not present in a modern loudspeaker. And most of the modern transfers suffer of the insistence in showing too much, including sounds that were not supposed to be heard. In this sense an older machine is more effective, as its limitations hide or attenuate the record faults. Having said that, I already ordered and am anxious to listen to the centennial King Oliver CD set that got great reviews - as I don't have any original source record, only 78 rpm dubs, I cannot experience spinning one of his Gennetts in an Expert.
Secondary to the music and the sound, there is the satisfaction of restoring or putting back into function the old machines - they are simple enough to allow restoration and maintenance even 100 years after their manufacturing, with new replacement parts machined or cast as needed, contrary to most modern electronic gear, that after 40 years become almost impossible to restore, with parts for which there is no replacement - just think of the displays that die out with time and the bespoke ICs and are in the heart of all hifi gear since the 90s.
There are also the aspects that are more common to all collectors, like the aesthetics, the completeness and the rarity. I admit that over the years I bought some machines and records for one of more of these reasons, but certainly none is a main driver for me as collector. Amassing every single variation of a model or a brand does not attract me - they will all sound the same anyway - but I open exceptions, like wood-case colonial models, that to me fit in all three categories: they are nice to look at, more or less rare and all their variants are interesting because diverse in shape and form.
Lastly, there is another aspect that all collectors share: the hunt. The best gramophone and record in my collection is the one I still don't have and is waiting to be added in a special place.