Sparse $4$-critical graphs have low circular chromatic number