• – add protocol for file
• – remove protocol for http if no web spidering is required

old line
   <value>protocol-file|protocol-http|urlfilter-regex|
      parse-(text|html|js|tika)|index-(basic|anchor)|
      query-(basic|site|url)|response-(json|xml)|
      summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
new line
    <value>protocol-file|urlfilter-regex|parse-(text|html|js|tika)|
      index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|
      summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>

Add a line to stop the root from being indexed

   <property>
      <name>file.crawl.parent</name>
      <value>false</value>
   </property>

2. Create Seed file. This contains a list of URI’s that will be starting point for crawler

• in Nutch home directory, create seedURLs

     > cat SeedUrls/url
     file://d:/Sandbox

3. Update conf/regex-urlfilter.txt

update line to allow file

 # skip file: ftp: and mailto: urls 
 #-^(file|ftp|mailto):  
 # accept anything else

+.

4. Clean data from previous runs

This step is important as every time you run nutch saves the crawled links in database and will start, indexing the saved links, irrespective of the new seed uri’s have the old link.

> rm -r $NUTCH_HOME/indexDir

5. Run Nutch

>  cd $NUTCH_HOME

>  bin/nutch crawl seedUrl/urls  -dir indexDir

• Ease of use, installation and maintenance
• Schema free – yet support searching, indexing, CRUD operations
• Support storing or multiple data formats, including large binary data
• Small footprint
• Scalability
• Cost
• High performance by way of lesser functionality and dropping

noSQL is to databases what RISC architecture are to CPU architecture. Performance and simplicity over complexity.

NoSQL style databases has often been termed non-relational databases, not surprisingly noSQL databases support relationships.  If they did not support relationships they would be of no use.  noSQL databases is a movement towards schema free databases.  They do not support implicitly support JOIN, this operation is done within application  using iteration or hookups.

Whats wrong with Relational Databases

• Cost
• Setup and configuration effort
• Supervision and periodic tuning
• Overkill for small applications and medium sized apps
• Rigid schema, applications grow and schema soon starts getting less cleaner
• Modifying schema is hard
• Harder to scale
• Most web applications geared towards search/insert
• Normalizing data:
  • was important when storage was costly
  • Joins are costly
  • Supports data integrity where rules are strictly enforced and data is consistent
  • hard to maintain multiple levels of truth
  • Not suited for fuzzy search or non-indexed search
  • De-normalized data is better manipulated by application where rules can be customized depending upon the situation
  • De-normalized data is better approach if application only does inserts and searches
• noSQL databases need atomicity on a single record level only

Drawbacks of noSQL databases

• Data Integrity needs to be enforced in application
• Duplicate and inconsistent data

Advantages

• Small footprint – mobile devices, web based apps
• Suited for JSON/Rest based interfaces
• Scalability
• Performance
• Availability
• Infrastructure – supported in clouds, any OS
• Fluid schema

Both these DB are disk based and document oriented Open-Source databases.  They have a rich text based administration interface

Useful Links

MongoDB Admin

MongoDB interactive shell

NoSQL Landscape

MongoDb

Concepts/Terms MongoDB==Relational DB

• A collection may have heterogeneous set of documents
• No need to pre-define columns or fields within a collection

Drawbacks

MongoDB – Installation and First Steps

MongoDb Interactive shell – basic commands

MongoDb Interactive shell – searching records

Searching is just one of the areas where we will see the influence  of Web 3.0, changes will also affect on how we are able to mashup information from different sites to get relevant answers.  For example a query ‘list all US Presidents that studied in ivy league ?’ .  looking at the query we instinctively know what we are looking for, search engines will parse it for keywords and list out all web pages that have the words US president, ivy league.  and since the term ivy league applies to certain universities the results would be all over the place.  The above scenario is a bit simplistic because most search engines understand that ‘US Presidents’ as a single keyword and not two, anyway I hope you get my drift on what we can expect from Web 3.0.

Semantic web article is also posted at informationzen.org.

• hCard for user profile
• hCalendar for calendar or events
• hReview – for product/recipe reviews
• geo – geographical location
• hatom – rss entry/blogs/web pages
• XFN – human relationships

Using microformats is not just useful for semantic web, which is not expected until year 2010, but embedding microformats is more likely to increase your rankings in search listings.  With the amount of data on web growing exponentially, keyword based search is becoming less of a solution.  Yahoo has already announced that it will be processing web content for semantic data and its support for microformats.