Algorithms, Design, Code and more: June 2013

Solr has REST api's available for performing various searches on indexed documents. The client generally issues GET requests to Solr with different parameters (fields, row, facet, etc.) set. Since there typically are size/ query length limitations on GET requests (imposed by container, OS, etc.), Solr allows the same queries to be issued to the Solr RequestHandlers as POST request as well.

We ran into one such issue with long GET request to Solr from SolrNet and did a few changes to solve the same.

Solr Side Changes:
First up, we increased the headerBufferSize of the application server as explained on SO here and increased the maxBooleanClauses parameter in solrconfig.xml. This allowed Solr side to start responding to much longer GET requests. The problem however wasn't solved. The client side was a dot net application running within IIS having additional length limitations imposed by Windows OS & the dot net framework.

SolrNet Side Changes:
In round two, we went for a better fix and switched over to a POST requests in place of long GET requests. The solution is largely the same as mentioned on the SolrNet group here & here. The difference being to switch over to a POST request from within the Get() method of the SolrConnection.cs class, when the request string is longer than a configurable threshold value.

public string Get(string relativeUrl, IEnumerable> parameters)
{
string st1 = GetQuery(parameters);
if (isQueryLong(st1)) {
var bytes = Encoding.UTF8.GetBytes(st1);
try {
using (var content = new MemoryStream(bytes))
return PostStream(relativeUrl, "application/x-www-form-urlencoded", content, null);
} catch (WebException e) {}
}
else{
// Do it the normal way via GET request
}
}

Update: PostSolrConnection.cs class has made it to the head branch of SolrNet.

Algorithms, Design, Code and more

Monday, June 10, 2013

Solution for making Long GET Request to Solr via SolrNet