robotsMap memory continues to grow

[ZigHuang]

Hello, I am using colly to visit some websites and set c.IgnoreRobotsTxt = false.

As it runs, you will observe that the memory continues to grow over a relatively long period of time.

This growth rate is difficult to observe by using pprof.

As a control experiment, I set up a set of colly with the same configuration, the only difference was setting c.IgnoreRobotsTxt = true.

After running for a period of time, the memory of the latter continues to be stable within 1G, but the memory of the former continues to increase.

I can't find any other way to reset this robotsMap if c.IgnoreRobotsTxt = false is set other than reinitializing via colly.NewCollector()

[ZigHuang]

I can raise a PR to set the size or other limits of robotsMap to avoid continuous increase in memory.