I like my neighbor as more than a friend, but he's not sure if he likes me the same way?

He's 6 years older than me. We ride horse together all the time and he seems to like me, but when people get him alone and ask him if he likes me, he says, "I don't want to ruin anything between our families," and "I live with my brother. I want to wait until I have my own house." But now his brother moved out to live with his girlfriend. My family adores him and I feel that we are ready to be more than just family friends. Another thing is that he's ex-Amish, so he's kind of submissive and quiet to some people. He's opened up to me and I know he trusts me. I don't know what the next step would be for us. I've never had a boyfriend and I don't think he's ever had a girlfriend. I don't want to miss what we could have. I'd appreciate any input! Thank you for your time!