Comment by h3ctic

I disagree, if you realize: "we need to refactor this file/module/class/etc." then it becomes a new task. Or more general, "we need to refactor the organically grown architecture": it's a new task. Working on a task should not prohibit your thinking about additional tasks, just add them to the backlog.