When I think "Digory" I think "The Chronicles of Narnia." I don't have any strong feelings one way or the other... I just think about magic rings, and lions, witches and wardrobes. And I thought Hickory, but with a D... :-| ... and other bad sex ed jokes.
Gwyneth Paltrow named her baby Apple, so hey, do what you like. But in addition to what Rob said, also think about any possible aggravation the kid may endure throughout life.
My family is from Côte d'Ivoire, so I have what is considered to be an unusual name in the US. Kids made fun of it. Teachers butchered it. College professors always twisted their faces before attempting it. Announcers struggle with it when I lecture, or get an award, and at every graduation. Even people I know don't get it 100% correct, but I'm tired of correcting people so I let it slide. Unless people know me, hear my voice or see my picture, they don't know what gender I am until we meet. I always worried about what people thought of it and me when I applied for anything or submitted a resume (which eventually led to my starting my own firm, and that's been a good thing). When I meet new people, I have to have the same tired conversation about my name that I've been having the last 3 decades. (No, Côte d'Ivoire is in West Africa. Next to Ghana. My name means blah blah blah. ... but I really want to say, "Leave me alone." That gets really old and just reminds me that I'm "out of place.") Yes, it's unique... but do I really need to be unique at 3am in the 24-hour supermarket? I think not.
... Anyway, something to think about from someone who has an "unusual" name... but I love it because it reminds me of my heritage and has meaning for me.