phpSERA: Search Engine Ranking Analysis

Latest version of phpSERA is 0.3alpha1 and was released on 2004-05-04.

This project has been ON HOLD since 2004-05-04. The 0.3alpha1 code is stable enough to use, but until the project is revived, you will need to maintain your own regex's.

Overview

This is phpSERA, a project which primarily aims to create a free tool for analyzing search engine rankings for websites.

And we need you! Add support for any search engine yourself and contribute it to phpSERA (get credits). This should help make phpSERA not only a free, but also a true alternative for the expensive commercial tools.

phpSERA v0.3alpha1 available for download
phpSERA's Source Forge project page

Download

You can download the phpSERA-0.3alpha1 from SourceForge!

Changes since 0.2final:

now 100% support for UTF-8
indexed page is now stored and displayed with each ranking (i.e. '/products.html', '/docs/')
added support for HTTP proxy (see global.inc.php)
added support for custom User-Agent header (see global.inc.php)
added textarea to Keyphrase Management to add multiple phrases at once

Pattern updates

Current update (ONLY for phpSERA-0.3alpha1 and up): searchengines_2004-05-04.sql
Previous update (suitable for 0.2final): update_2004-01-26.sql

As with all ranking analysis software, it's all about keeping and maintaining support for a list of search engines. In the last months of 2003 I tried to maintain support for about 100 search engines, but I can't handle it on my own. Therefore, I reduced the number of supported search engines to about 50, which I will be able to maintain for all phpSERA users out there.

The update_yyyy-mm-dd.sql file has been created with mysqldump:

mysqldump -t phpsera searchengines |grep -v ^# > searchengines_<yyyy-mm-dd>.sql

And was manually changed to replace the INSERT commands with REPLACE commands, so it will either UPDATE or INSERT based on the existance of a record with the same ID.

perl -pi -e 's/^INSERT/REPLACE/g' searchengines_<yyyy-mm-dd>.sql

Installing the update

You can update your search engines table with this command:

mysql phpsera < searchengines_<yyyy-mm-dd>.sql

Screenshots

These are screenshots from 0.2rc5-rc9. No big changes to GUI since then.


Home	New Report (1)	New Report (2)


Quick Ranking	Overview (1)	Overview (2)


Trend Analysis	Search Engine Management

phpSERA's interface language can be set to German (thanks Markus!), English and Dutch. I'm developing it using English, Dutch, German, Italian, Spanish, Swedish, Finnish, Norwegian and Danish keyphrases and search engines.7

How it works

Given one or more search engines, one or more keyphrases and a single URL to look for, phpSERA connects to the search engine(s), parses the result pages and displays a ranking report. Parsing is done by stripping the header and footer of result pages using quick'n'dirty regular expressions (frequently updated and available from this website). Finally, the result list is split using a given separator and a ranking (or error code) is returned.

Currently only the first result page is parsed! Fortunately, Yahoo.com, Voila.fr, Sol.se, Lycos.nl, Track.nl, Vinden.nl, all versions of Google and AlltheWeb and many other SEs allow displaying up to 40 or 50 results on a single page. QualiGO.de/.at/.ch/.nl, Eniro.dk and Orbis.dk display 20 results by default. All local versions of MSN display 15 by default, as well as Freenet.de. Virgilio.it is able to display 20.

Supporting multiple result pages is on the phpSERA wishlist though (see "Wishlist" below).

A regular expression to recognize the beginning of a result list
Example for Google.com:
```
<\/table><div>
```
Example for MSN.com:
```
<li>
```
A regular expression to recognize the end of a result list
Example for Google.com:
```
<\/div><br clear=all>
```
Example for MSN.com:
```
<\/ol>
```
A separator string
Example for Google.com:
```
</a></font>
```
Example for MSN.com:
```
<li>
```

Compare

Feel free to try any commercial alternative; there are many tools available. The main advantages of phpSERA are:

phpSERA is free, open source, and you may do with it whatever you want (under GNU GPL, that is)
phpSERA operates serverside, and therefore won't consume as much bandwidth at the client
phpSERA has a high potential, depending on the quality of the released code and frequency of updated search engine patterns (possibly contributed by you?)

Supported Search Engines

Most major and several regional search engines are supported by phpSERA. phpSERA uses regular expressions to determine the rankings and it's easy to add other search engines: all you have to do is a come up with a short regular expression that's adequate for parsing the HTML output for that particular search engine.

Quick fact: on May 2nd 2003, I was able to add support for 60+ search engines during a single working day!

List of supported search engines, as of May 4th 2004

In the table below, search engines which have the Language set to @@ are considered 'international' search engines. Most of those search engines have a default setting of 'search the whole web' rather than 'search within (English|French|Dutch|German|...) pages'. phpSERA assumes you want to know your rankings regardless of the language of the keyphrase, e.g., you'd probably rather know what ranking you have when John Doe searches for 'widgets', rather than the ranking you have when John Doe specifically searches within a certain language or country (as opposed to regional search engines, obviously).

ID	Name	Language
31	AlltheWeb.com (worldwide)	@@
33	AlltheWeb.com IT	it
95	Altavista AT	de
84	Altavista CH (French)	fr
85	Altavista CH (German)	de
88	Altavista ES	es
87	Altavista FR	fr
86	Altavista NL	nl
48	Excite (worldwide)	@@
9	Fireball DE	de
26	Freenet DE	de
43	Freeserve UK	en
1	Google (worldwide)	@@
77	Google AT	de
78	Google CH	de
12	Google DE	de
100	Google DK	da
13	Google FI	fi
14	Google FR	fr
15	Google IT	it
11	Google NL	nl
122	Google PL	pl
121	Google SE	sv
5	Ilse.nl	nl
118	Interia PL	pl
103	Kvasir NO	no
106	Libero IT	it
18	Lycos (worldwide)	@@
75	Lycos AT	de
76	Lycos CH	de
17	Lycos DE	de
55	Lycos FR	fr
65	Lycos IT	it
3	Lycos NL	nl
27	MSN (worldwide)	@@
72	MSN AT	de
73	MSN CH	de
25	MSN DE	de
61	MSN DK	da
60	MSN ES	es
62	MSN FI	fi
56	MSN FR	fr
57	MSN IT	it
2	MSN NL	nl
59	MSN NO	no
58	MSN SE	sv
107	Netsprint PL	pl
119	Onet PL	pl
52	Sonera Plaza FI	fi
64	superEva IT	it
117	Szukacz PL	pl
97	Tiscali IT	it
7	Track.nl	nl
4	Vinden.nl	nl
63	Virgilio IT	it
44	Voila FR	fr
8	Yahoo (worldwide)	@@
16	Yahoo DE	de
20	Yahoo DK	da
68	Yahoo ES	es
54	Yahoo FR	fr
6	Zoek.nl	nl

Limitations

Parser limitations

phpSERA only works on search engines that actually output HTML pages containing the complete result list. Most (Google, Lycos, Excite, MSN, AlltheWeb, ...) do, but some regional search engines use complex Javascript/DHTML document.write, redirect and loading structures which confuse phpSERA. If you know any of these, please don't hesitate to contact me so I can list them on this site.

Wishlist

automatic detection of layout changes in search engine result pages, webmaster notification
support multiple result pages
support for SEs like "T-Online.de", which display a redirect link in their resultpages (and not the currently required direct link)
W3C XHTML compliant interface

System requirements

Any system running PHP/MySQL should be able to run phpSERA, assuming the webserver is allowed to set up outbound connections to the search engine webservers (e.g. google.com:80). The webinterface has only been tested using Internet Explorer 5 and 6, but Firefox seems to work fine. Most browsers should work, as long as they support Javascript.

License

phpSERA is licensed under GNU GPL. Therefore, you can do with it what you want, even use it for commercial purposes, as long as any changed code is released under the GPL again. The search engine regex's are free to use, but cannot be made part of any kind of proprietary software.

Development status

[May 5th 2003]: project approved by SourceForge
[May 7th 2003]: phpsera v0.2rc1 pending
[May 12th 2003]: phpsera v0.2rc1 available for download
[June 23th 2003]: phpsera v0.2rc3 available for download
[July 19th 2003]: phpsera v0.2rc4 available for download
[July 24th 2003]: phpsera v0.2rc5 available for download
[July 29th 2003]: phpsera v0.2rc6 available for download
[July 31st 2003]: phpsera v0.2rc7 available for download
[August 21st 2003]: phpsera v0.2rc8 available for download
[September 24th 2003]: phpsera v0.2rc9 available for download
[January 26th 2004]: phpsera v0.2final available for download
[May 4th 2004]: phpsera v0.3alpha1 available for download

You can view the current documentation in Rich Text Format: readme.rtf

Credits

Thanks for all your efforts!

SourceForge for their outstanding services
Leander van Hoesel (tester)
Barry Haanstra (tester)
developers of jpGraph (used for graph generation, though still figuring out how to use it)
developers of ADODB (used for DB connectivity)
Markus Niklaus (PHP code contributions, improvements and German language support)
Robo (bugfix in overview.php)
Stephan Häuslschmid (improvements to the trend analysis)

Donations

Thanks for supporting this project!

Matt Snyder (July 2004)

(100% of your donation will go to the Electronic Frontier Foundation, the nonprofit organization fighting for your rights on line)

Press

Internet World 05/04 (German edition, installation guide on p80-82)
International PHP Magazin CD 06.2003 (phpSERA-0.2rc9 included on cover CD)

Contact me

You can contact me through this page.

Official home of phpSERA: http://phpsera.sourceforge.net/

This page was last updated at 10-08-2008 07:05 UTC

phpSERA - free search engine ranking analysis software for you