rag +horn, describing the animals as non-prime trophy specimens, i.e. "rags". Another explanation is that it is descriptive of how the new antlers have a velvety covering that sloughs off each year and can hang like rags.
(US,informal) Any male elk (bull) with antlers between one and six points, non-inclusive, on either side. Also called an "intermediate bull" (bulls with two, three, four or five points on either side).