I'm sure this is an issue you have pondered for the last 3 years and I certainly can understand your concern about wanting to have the "appropriate" response to the question.
First, I would encourage you to flash forward and imagine your son being 13 years - 18 years old and what the ultimate message will be that you give your son assuming you have no further contact with the father. Only you can decide what that would be, but I would encourage you to frame it as pleasantly as possible; his father is a part of who he is so self esteem will be connected to his perception of his father. "Your father was a very kind man. He wasn't able to make a commitment to have a family at that time, but I am so happy he gave me you." You will likely be heavily questioned in the future and rather than focusing on the negatives, I would encourage you to let your son know that you actually don't know how his father "turned out" and he may be a wonderful person and you only want to speak of him in a positive manner. There's a likelihood, in time, with technology as it is it will be possible for him to find his father if he wishes. If so, he can let his father explain more about himself. Therefore, my suggestion would be coming up with a description you are happy with and sticking with that. It's impossible to know if his father will attempt to reconnect in the future and again, if this happens he will be able to fill in any gaps.
Back to a three year old. Based on the general message you decide will be your description of his father (i.e. you will want to function within the perimeters you set yourself in that response) he needs to hear messages that:
1. Do not provoke any anxiety
2. Encourage him to feel whole, and loved and like any other family
3. Normalize his situation
For now, I would consider saying something to the effect of, "Not every family has a mommy and a daddy. Some have just a mommy or just a daddy or just a grandpa or just a grandma, etc. I am so lucky to have you and we are lucky to have each other." "Lots of families only have a mommy or a daddy. Our family is me and you." "The important thing is that we have each other."
The words Dad, Daddy, Father, etc. are loaded. Be careful how you use them. I tend to believe those terms are held out for the person that actually is involved in their children's lives and when children use it that do not know their mother or father it is somewhat fantasy based and gives you a glimpse into their wishes to have a "daddy." In this situation here is an example, "Mommy cared about a man that wasn't ready to have a family, but I was very very lucky and he helped me be able to have you even though he has never met you." as you ponder exactly what to say, I would be careful not to create a fantasy, by your explanation, that has him thinking that "Daddy" might walk through the door.
I would encourage you to get involved with other single parent families to normalize the way he perceives your family. That way he can see that lots of families are made up of different people. There isn't always a mommy AND daddy.
Also, the more positive male role models he has in his life the better. It can give him some healthy balance as he matures, and as I always say, single parents must sometimes make the village that will help them raise their children. Positive, supportive people in your lives that can help out when needed and give him other positive relationships with adults can be a gift for you and him.
As he gets older you can begin to explain that there is a difference between fathering and being a father while still communicating respect about his biological father.
Age appropriate literature is always effective in helping to normalize situations. Here are a few titles you may find helpful: Do I Have a Daddy?: A Story About a Single-Parent Child by Jeanne Warren Lindsay and Jami Moffett (Paperback - Nov 1, 1999); Love Is a Family (Hardcover) by Roma Downey (Author); The Family Book (Hardcover) by Todd Parr (Author).
While I know I did not give you a concrete answer, I hope this guidelines give you some direction as you decide the message you want to give your little guy.
All the best to you and your future with your son,
Becky Aud-Jennison, MA, LCPC