Discuss mailing list

[Martin C <martin AT mchomenet.com>]

Hi Calum,
I've been working on a search engine too, started just before writing the 
article that was discussed here 2 weeks ago.

> The search system is http://www.sphider.eu/ - a PHP/MySQL search
> system that is a bit outdated. In the future I plan to move to
> something a bit more sophisticated, but at the moment it seems to do
Mine has the initial search page in HTML, which sends off a query to the main 
search engine which is written in C and called via CGI. I currently use 
SQLite3 for data storage. It is written from scratch.

> the job. The database kept crashing during indexing, but after adding
> some swap space to the server, the problem seems to be fixed.
My spider is written in PHP, and I noticed that after a long time of 
indexing, PHP would complain about a memory error. I suspect a small memory 
leak somewhere. I think I need to tweak php.ini a little. I intend to 
re-write it in C in the next few months.

I have over 80,000 sites currently indexed, and it is OpenNIC aware. My 
system concentrates on the META tags (keywords and description) with a 
fall-back to first 300 characters of a page's content if it can't find a META 
description.

Who knows, maybe we can combine resources and know-how?

Thanks to Peter for the lead-in to my reply, he has been testing my search 
engine nearly since the start.

Let me know if you'd like a link to it to have a look-see.

Martin.