If you do bad things, people can't just pretend that never happened. However, you can work hard to surpass your infamy with righteousness. Likewise, if a valiant hero turns bad he should be able to use his fame to rally people up or something and slaughter if he so chooses just because that's where he wants to go as a character.
That my friends, is good plot. but the current rules break that scenario from ever occurring.
For guildless mages I think they should be able to build either fame or infamy at the same time and therefore if they have high in both calibers it'll open 'real' neutral doors story-wise. It makes more sense to double-lap fame and infamy than to take it away and deprive someone of fame or infamy in general. Because currently, losing it instead of building it keeps Guildless mages from achieving a big name in the world unless they go one way or another.
If you ask me, I think this should be acknowledged to some degree as I was pondering it myself and just found it better for story and system continuity all together. What if you're S-rank with only 400 fame because you did all bad and good jobs and instead of growing off it you had to clear your name and didn't get anything back?? As we all know, you can't take quests you already took. Therefore, this is breaking the fame system if you ever wish to switch sides. Or breaking the mission system if you ever wish to go Green Ranger to White Ranger lmao.