The expression day job is often used for a job one works in order to make ends meet while performing low-paying (or non-paying) work in their preferred vocation. Archetypal examples of this are the woman who works as a waitress (her day job) while she tries to become an actress, and the professional athlete who works as a laborer in the off season because he is currently only able to make the roster of a semi-professional team.
While many people do hold a full-time occupation, "day job" specifically refers to those who hold the position solely to pay living expenses so they can pursue, through low paying entry work, the job they really want (which may also be during the day). The phrase strongly implies that the day job would be quit, if only the real vocation paid a living wage.
The phrase "don't quit your day job" is a humorous response to a poor or mediocre performance not up to professional caliber. The phrase implies that the performer is not talented enough in that activity to be able to make a career out of it