Restricting tokens due to the search context is better done in the generic search part instead of repeating the same test in every tokenizer implementation.