This script looks for the wikipedia network around a certain topic.  It first finds the outlinks from and inlinks to that page in the wikipedia.org domain.  It will use the common list as startingpoints and calculate the nr of pages that link to each page thus found.  You will notice this has a long tail distribution. For all pages with an nr of links > the 'long tail cut-of point' their outlinks are retrieved and the shared links are calculated again for that set of links.  This proces is repeated for crawlDepth -1 times.<br/><b>Warning:</b> crawling is exponential!  More and more links are found each time.  A page is fetched once each second.  Even a small network takes a looooong time.<br/><b>Tips:</b> exclude dates and categories.  ctrl-f on iteration to get to the results quickly<br/><b>Note:</b> Notice that only for getting the first startingpoints 'what links here' is used.  We might wanna use it for the other pages as well - instead of just counting shared (out)links.  In doing so we would get shared (out)links of which we are sure they also have an inlink to a previous iteration.  The implications of both methods need to be thought through.<Br/><hr/><form name='' action='' method='POST'>wikipedia url: <input type="text" style="width:80" name="wikipediaUrl" value=''>
<br/><table><tr><td>do not include these pages in the network (1 url per line):<br/><textarea rows="8" cols="80" name="exclude">http://en.wikipedia.org/wiki/Non-profit_organization
http://en.wikipedia.org/wiki/Charitable_organization
http://en.wikipedia.org/wiki/501%28c%29#501.28c.29.283.29
http://nl.wikipedia.org/wiki/In_het_nieuws
http://nl.wikipedia.org/wiki/Wikipedia</textarea>
</td><td align='left' valign='top'>also remove: <Br/>(multiple select possible)</td><td><select MULTIPLE size='8' class="filt_fieldselect" name="alsoRemove[]" onChange=""><option value="dates" selected>dates</option>
<option value="categories" selected>categories</option>
</select>
</td><td>links that are automatically removed: non wikipedia urls, User:, Talk:, Special:, Portal:, Image:, Help:, Main_page, Wikipedia:, Template:, &action=, /w/index.php (+ their dutch equivalents) </tr></table><br/>depth of crawl: <input type="text" style="width:3" name="crawlDepth" value='1'>
<br/>cut of long-tail (in each iteration): +-<input type="text" style="width:3" name="cutTail" value='8'>
%.  This will only use those page with the top x% of inlinks - with the modification that it will keep pages with the same amount of inlinks from the page where it's supposed to be cut.  Note to myself: find a good cut-of point based on the actual distribution of the links (see <a href='http://en.wikipedia.org/wiki/Long_tail#Probability_distributions_with_.22long_tails.22'>probability distributions with long tails</a>)<Br/>show removed links <input type="radio" name="showRemoved" value="0" checked> no
<input type="radio" name="showRemoved" value="1" > yes
<br/><input type='submit' value='whateva!'><br/></form>last update 13/2/2007: added dutch pages to be removed, fixed whatlinkshere to be universal<Br/><hr/>