On repair (fast) days I wouldn't worry about it - there's loads of folks doing zero calorie liquid fasts with no harm done. On feed days you *do* need to try to eat to your TDEE, if possible.
Are you low carbing? Ketosis often means one doesn't feel hungry (it's actually a free semi-reliable way of telling whether one is *in* ketosis). This is good - you're fat burning. Yay! But, as you're finding, one sometimes has to make oneself eat! Maybe looking at food porn and cooking extra-special stuff would help you? Failing that, when you do eat make sure that the foods are *really* high density calories, e.g. cheese, nuts and the likes, lots of olive oil... All the best, FatDog.