phpSERA
Matthijs Koot, 22th July 2003

Development notes v0.2

 phpSERA-0.2rc4  
Some essential GUI elements are provided in form of phpMyEdit generated scripts:
-	admin keyphrases
-	admin reports
-	admin reportrules 
-	admin websites
(although they will be customized before the first full release of phpSERA)

Description
phpSERA is a webbased application for the analysis of ranking positions in search engines worldwide.

Cool features
-	supports 88 search engines, divided in English, Dutch, French, German, Spanish, Italian, Finish, Norwegian, Swedish, Danish and global search engines
-	monthly ranking reports
-	smart analysis: phpSERA only performs analysis if keyword and search engine language match (except global search engines ofcourse, for which analysis is performed regardless of keyphrase language), saving both system and network resources
-	parsing through agile regex’s guarantees basic robustness
-	easily debug and add new search engines, automate analysis through cron, etcetera
-	per search engine ranking trends (uses JpGraph by Johan Persson)
-	it’s free, open source and under development

Introduction
The travelling-related company I do some parttime work for develops and maintains several websites, including Eurocampings.net. The website - a camping search engine - supports 10 languages. We wanted to increase international traffic by means of Search Engine Promotion (SEP), which in turn meant we needed to improve and monitor our rankings. phpSERA (merely) aids in the latter. 

How to use phpSERA

1) add a website
2) add keywords you want to monitor
3) start your first analysis

Ofcourse, you may add search engines first (optionally). Each keyword is assigned one of the following languages:

 EN	English (default)
 NL	Dutch
 DE	German
 FR	French
 ES	Spanish
 IT	Italian
 DK	Danish
 NO	Norwegian
 SE	Swedish
 FI	Finnish
Note that keywords and websites are immutable. This measure is taken to prevent corruption and inconsistancy of previous ranking reports. 

By default, the keyword language and search engine language have to match in order for phpSERA to perform an analysis for a certain combination. This is to avoid unneeded bandwidth usage (and possibly prevent you from being banned/blocked by a search engine for performing too many searches). With the sole exception of general, worldwide search engines, which have the language code “@@”. For those, ranking analysis is performed with any keyphrase, regardless of the keyphrase language. (see Appendix A)

International Search Engines

For international oriënted users of phpSERA (like myself), I recommend you to Keep It Stupidly Simple. It is easy to loose sight with all them search engines out there, especially with the many SE partnerships and takeovers. 

Use phpSERA by first adding all keywords you want to monitor.  Optionally, add more search engines (or contact me for help). Make sure you have assigned the right language to each keyword and search engine.

Then just have phpSERA periodically analyze the result pages from search engines for ALL keywords. 

Future improvements
- add a simple spider (similar to simspider on searchengineworld.com) to use as a basic reference for web developers
- add a live debugging of SE settings (ie regular expressions) to optimize the parsing process

Application functionality

Full Analysis
SERA must be able to produce and store reports of the ranking positions of a website for one or more selected keywords, in one or more selected search engines.

Quick Check
SERA must be able to produce quick, temporary (non-persistent) reports of the ranking position of a single, user-inputted URL for a single, user-inputted keyphrase, in one or more selected search engines.

Report overview
SERA must be able to present a listing of all available ranking reports.

Show Report 
SERA must be able to show all data from a selected ranking report.

Trend Analysis
SERA must be able to present a comparitive graph (or table) for the rankings of a selected website, for one or more selected keywords, in a single selected search engine.

Administrative functionality

Keyword management
- add 
- update (ID and phrase are immutable)
- delete (constraint 3)

Website management
- add 
- update (ID and URL are immutable)
- delete (constraint 1)

SE management
- add 
- update (ID is immutable)
- delete (constraint 2)

Report management
- add (impossible)
- update (impossible)
- delete (constraint 4)
- NO support add, update, delete for individual reportrules

Data Constraints

1) Deletion of a website: 
Delete all related records from WEBSITES, KEYWORDS, REPORTS, REPORTRULES

2) Deletion of a SE: 
Delete all related records from SEARCHENGINES,  REPORTRULES

3) Deletion of a keyword: 
Delete all related records from KEYWORDS, REPORTRULES

4) Deletion of a report: 
Delete all related records from REPORTS, REPORTRULES 

Current limitations
Many commercial ranking analysis tools experience most of these problems too!

1) No user administration, authentication
Beyond the scope of the project. Please use .htaccess and ACLs for that :-)

2) The rather weak parsing algorithm 
The core parsing algorithm assumes that your ranking position can be determined by the presence of a domain name in a search result list item. At least two caveats: 
•	a domain name may be part of the META description - or whatever is shown in the by the search engine (SE) in it's result list - of a competitive, higher positioned website, resulting in an incorrect analysis (too optimistic); note that the first occurance of your domain name is the cue to return a ranking position
•	SE's hiding, obscuring or simply not showing domain names anywhere in the search result, cannot be analyzed by phpSERA (including some SE's using redirect links)
•	Changes in the (HTML) layout of a search result list will almost always result in changing the regular expressions for that SE. Periodical updates are needed and unavoidable, until search result lists are standardized or at least less prone to change (XML). Fortunately this can be partly prevented by the use of agile regex’s in the [searchengines] table.

3) Only the first resultpage is parsed
•	Google and many other SE's allow to display N results on each page, in which case there is no problem
•	if you're only interested in top 10-20 results, this might not be a problem for you as all SEs display 10 to 20 results on each page
•	phpSERA focusses on the top 20, max. top 50 results

4) Not suitable for complexity
There is no support for analysis of how well any specific page of a given website does. Currently phpSERA only seeks for the first occurance of the URL of a website, and returns that as ranking position.

5) Paid rankings
There is no support to distinct between 'paid high rankings' and normal results if there is no actual difference between them in HTML code. Haven't seen this anywhere yet, for that matter.

Appendix A
Supported Search Engines, as per 22 July 2003.

Note that if Language is “@@”, it’s a high traffic international search engine which is used by people worldwide with keywords in their own language, while that engine’s default setting is to search the WEB rather than pages in that specific language. I.e. probably you too use Google.com most often in it’s default setting (“Search the web” rather than “Search for pages from (…)”).  

For those high traffic engines, we assume you want to know your rankings regardless of the language of the keyphrase.

Title	Language	
1klik DK	dk	
Advalvas BE	be	
AlltheWeb.com DE	de	
AlltheWeb.com DK	dk	
AlltheWeb.com ES	es	
AlltheWeb.com FI	fi	
AlltheWeb.com FR	fr	
AlltheWeb.com IT	it	
AlltheWeb.com NL	nl	
AlltheWeb.com NO	no	
AlltheWeb.com SE	se	
AlltheWeb.com UK	en	
Altavista AT	de	
Altavista CH (French)	fr	
Altavista CH (German)	de	
Altavista ES	es	
Altavista FR	fr	
Altavista NL	nl	
Austronaut AT	de	
Austrosearch AT	de	
Bluewin CH	de	
Eniro DK (=SOL.dk)	dk	
eTools CH	de	
Excite AT	at	
Excite DE	de	
Excite ES	es	
Excite FR	fr	
Excite IT	it	
Excite UK	uk	
Fireball DE	de	
Freenet DE	de	
Freeserve UK	uk	
Google (worldwide)	@@	
Google AT	de	
Google CH	de	
Google DE	de	
Google FI	fi	
Google FR	fr	
Google IT	it	
Google NL	nl	
Ilse.nl	nl	
Jubii DK	dk	
Lycos (worldwide)	en	
Lycos AT	de	
Lycos CH	de	
Lycos DE	de	
Lycos FR	fr	
Lycos IT	it	
Lycos NL	nl	
Metaspinner.de	de	
MSN (worldwide)	@@	
MSN AT	de	
MSN CH	de	
MSN DE	de	
MSN DK	dk	
MSN ES	es	
MSN FI	fi	
MSN FR	fr	
MSN IT	it	
MSN NL	nl	
MSN NO	no	
MSN SE	se	
QualiGO AT	at	
QualiGO CH	ch	
QualiGO DE	de	
QualiGO NL	nl	
SOL ES	es	
SOL NO	no	
Sonera Plaza FI	fi	
Suomi24 FI	fi	
superEva IT	it	
Tiscali CH	de	
Tiscali IT	it	
Tiscali NL	nl	
Track.nl	nl	
Tricus AT	de	
Tricus CH	de	
Vinden.nl	nl	
Virgilio IT	it	
Voila FR	fr	
Web DE	de	
www.fi FI	fi	
Yahoo (worldwide)	@@	
Yahoo DE	de	
Yahoo DK	dk	
Yahoo ES	es	
Yahoo FR	fr	
Zoek.nl	nl