Discuss mailing list

[Martin C <martin AT mchomenet.com>]

On Tue, Jan 28, 2014 at 07:35:27PM -0700, Jeff Taylor wrote:
>    Martin, something that might help you out... When I run any spiders I do
>    it in 3-hour increments.  Let them go for a bit, then call them to wrap 
> up
>    and close out nicely, then restart again at the 3-hour mark.  Not only
>    does it help flush out any memory leaks, but if for some reason the 
> spider
>    crashes, it will be restarted within a few hours anyway.

Thanks Jeff, I'll keep that in mind when I start it up next.

It'll be a way to make the spider run more efficiently, which is always a 
plus.

Martin.