Document ContextPipeline

[janbuchar]

• Follow up to refactor!: Introduce the ContextPipeline abstraction #3119
• We should explain how ContextPipeline works and how it can be used to create new crawlers or extend existing ones
  • best practices for writing crawlers
    • role of the contextPipelineBuilder
      • in custom subclasses of Crawler classes, it might make sense to not accept it and make the interface simpler
    • it is not necessary to be generic over Context, but any crawler should accept ContextExtension and ExtendedContext so that it supports extendContext in a type-safe manner
    • if the context pipeline middlewares delegate to methods of the crawler class, make them private
• error handling and non-failure pipeline interruption
• extendContext should also get a mention somewhere
  • including when to use extendContext (ad hoc enhancements during crawler instantiation) and when a custom pipeline makes sense (custom crawler classes)