Installing Canon LBP2900 Printer on Ubuntu 16.04

Capturing notes on installation of Canon LBP2900 printer on a 64-bit Ubuntu 16.04 (a first for this version) locally (usb). These are essentially a mix and match of instructions from the pages Canon CAPT Driver, installing canon printer on debian systems, & Canon CAPT driver in a sequence that works.

=> Pre-requisites:

 

- Cups installation:

(Note: These instructions are from the alternate open source foo2capt library library. I have retained the installs for now, did not apt-remove. Not sure if all of them are actually needed.

As such, the foo2capt code failed to build & install with a whole lot of other missing/ invalid dependency issues. The foo2capt project rewrite seems to be on, so for now dropped the idea of experimenting with it any further.)

- Work around to known CAPT 64-bit OS issues linking to 32-bit libraries:

=> Download & install the Linux CAPT printer driver:

Download Linux_CAPT_PrinterDriver_V260_uk_EN.tar.gz, also linked here.

- Untar & install the 64-bit > Debian packages: 

=> Add printer to system:

- Start/ Restart cups service:

 

 - Can be done via the command line using lpadmin:

  

- Alternatively can be done via System Settings > Printer:

Either type on terminal: 

Or via System Tools > System Settings & Search for Printer

At this point a new printer gets created in my system with the name "Canon-LBP2900-CAPT-English".

(NOTE: Important to use the port 59787 (and not 59687). Also note that in the /etc/ccpd.conf, port 59787 is mentioned as UI_Port (& used by the captstatusui to communicate), while port 59687 as PDATA_Port files. Once ccpd services have been started, you can telnet to check that these ports are listening. )

 

=>  Add printer to ccpadmin:

- Add "Canon-LBP2900-CAPT-English" to ccpadmin: (will override any existing/ old entry)

- Now there should be a proper entry corresponding to the printer "Canon-LBP2900-CAPT-English":

=> Restart ccpd services:

The ccpd services need to be restarted since they don't start up automatically, there could also be the one ccpd process issue.

 

=> View status of printer on captstatusui:

- In case you see a communication error, unplug your printer & plug it in again. On my system this works & the printer status changes to:
 "Ready to Print"

(Note: Steps for setting up capt rules for usb add/remove could also be tried out.)

=> Print test page:

 Next print a test page on "Canon-LBP2900-CAPT-English" & that's it for the set-up. 

 

One on Blockchain

The write-up on Hyperledger Fabric & other tools in the ecosystem has details on how to get off the ground. While the Hyperledger ecosystem is made up of several early stage technologies with a lot of ground to cover, yet it's got enough to interest technologists & blockchain enthusiasts.

Another blockchain project that looks promising is Corda. The Corda abstractions & concepts seem properly thought out, esp. given it's finance domain focus. Kotlin as the language of choice for Corda is another interesting read.

David's perspectives on Bockchain are insightful. Though, I don't much agree with his 3rd solution/ depiction to the problem (communal open database + trusted historians), it's similar to stuff that exists today. Corda's abstractions mentioned earlier seem better & might mainstream. His other point about building blockchains backed by databases is right. But specifically building one on Microsoft Sql Server seems weird, guess it's not even part of the Blockchain solution at Microsoft. Expecting to hear a lot more about rdbms & NoSql databases backed Blockchains in the near future.

Another much debated issue in the Blockchain community is around Block Size Limits. Different implementations have different size limits which can have an impact on the design of the Blockchain based application. Corda for instance offers an off-chain solution as attachments, while Ethereum potentially has no upper limit, though not recommended for storing large files.

Singapore

Discovery all the way through..

NUS, SoC, Changi, PGP, UTown, Overseas, Friends, Profs, Lab, Conversation, Biopolis, Garden Walks, City lanes, Metro, Bus, Campus, Merlion, Music by the Bay, Clarke Quay...

Research, PhD, Courses, Conferences, Seminars, Publications, Prez, RA/ TA, Summers, Docs++, DS, Stats, AI, ML, Foundations, Deep Learning, Vision, How small can we see?, AlexNet, NLP, Reasoning, Common Sense, Novelty, Minsky et al, ConceptNet, WordNet, Patholody, Medicine, NUH, Brain, CT, fMRI, Primates, Formal Methods, Papers n Papers n Papers...

A time in awe!

Fast Streaming Solution

High level view:

Web API          -> Kafka                     -> Storm (Streaming)

                    -> Hadoop/ HDFS    -> MR/ Hive (Batch)

 

Specifics TBD..

Lftp

Lftp a handy utility for all kinds of ftp, sftp, and other file transfer use-cases from the command-line on a *nix system. Give it a shot if there's ever a need..

Atlasian Stack

 - Jira
- Confluence
Among many others.. All well integrated enhance dev productivity...

Teradata

Teradata busy getting a chunk of the BigData pie. Teradata Parrallel Transporter (TPT) and Adv. SQL Combo makes querying Big Data sources fast and efficient using state of the art caching and other optimizations.

Getting Table Information From Hive Metastore

The Hive Metastore holds various meta information about Hive tables such as schema names, table names, partitions, fields, permissions, and so on. The metastore is built in a schema within a relational database, outside of Hive, and accessed via services by Hive clients. An embedded Derby database is configured as the default database for experimentation purposes, and must be changed over to something like MySql, Postgre, etc. for production use.

The Hive Metastore ER diagram is fairly straightforward. Once familiar with the schema, it is easy to query the metastore for information about the Hive tables. Here's a sample query to identify all Partitioned tables from a given Hive databases:

On Bitcoins

For now, the bitcoin gini index  is as divergent from the ideal as can be. More on what makes this tick to follow soon..

Hive Abstract Semantic Analyzer Hook

Hive allows Pre & Post Analyzer hooks to be added to the normal hive plan query generation flow via the AbstractSemanticAnalyzerHook class.

A custom hook needs to extend AbstractSemanticAnalyzerHook & override the preAnalyze or postAnalyze method as necessary.

Simple Sematic Analyzer Hook:
A sematic analyzer hook that logs a message each in the preAnalyze & postAnalyze methods, is shown below:

Configurations for Simple Sematic Analyzer Hook:

Urlencode and Urldecode in Hive using the Reflect UDF

Hive doesn't offer an in-built UDF to perform Urlencode or Urldecode. One option could be to write a custom UDF to fill for the void.

On the other hand, a rather straight forward alternative to have the same feature, as shown on the forum, us using the very generic Reflect UDF.

Hive Query Plan Generation

Hive query is passed through several built-in modules for the final plan to be generated.

The stages/ modules are:

Query 
     => (1) Parser 
               => (2) Semantic Analyzer 
                     => (3) Logical Plan Generation  
                           => (4) Optimizer 
                                => (5) Physical Plan Generation
                                         => Executor to run on Hadoop

Hive History File

Hive maintains a history of all commands executed via the hive cli. These commands are written to a file called .hivehistory, on the user's home folder.

Hive Optimizations

Explain Plan

Mappers and Reducers Count

Map Joins

Sorting

Optimization step:
Between the logical & physical plan generation phase of hive, hive optimizations gets executed. The current set of optimizations include:

• Column pruning
• Partition pruning
• Sample pruning
• Predicate push down
• Map join processor
• Union processor
• Join reorder
• Union processor

More on each of these optimizations to follow..

Build Hadoop from Source Code with Native Libraries and Snappy Compression

When running Hadoop using a pre-built Hadoop binary distribution (a downloaded hadoop-<Latest_Version>.tar.gz bundle), Hadoop may not be able to load certain native libraries. The following warning is also displayed at the time of starting up Hadoop:

"Unable to load native-hadoop library for your platform... using builtin-java classes where applicable "

This issue comes up due to the difference in architecture of the particular machine on which Hadoop is being run now vs. that of the machine on which it was orginally compiled. While most of Hadoop (written in Java) loads up fine, there are native libraries (compression, etc.) which do not get loaded (more details to follow).

The fix is to compile Hadoop locally & use it in place of the pre-built Hadoop binary (tar.gz). At a high level this requires:

Mocks for Unit testing Shell Scripts

1. Look at shunit for a sense of what kind of unit testing can be performed for Shell scripts.

2. For mocking up specific steps/ programs in the script, make use of alias.


Within a shell script testScript.sh this would be something as follows:

Mechanical Sympathy

A term that's gaining traction thanks to the LMAX architecture. Low latency applications running on the JVM need to be hardware gnostic to a large degree to be able to best leverage the computing power multi-core/ multi-processor architecture.

More details to follow soon on the topic out here, for the moment you could refer to Martin Fowler's post.

Real-time Face Reading

The machines getting better and better at face reading. Ancient mystics have another reason to worry. Won't be long before recommendation engines of various kinds get built that leverage this sort of technology.

More about algorithms in this space to follow..

Precision and Recall

Terms popular within search and Information Retrieval (IR) domains.

Precision: Is all about accuracy. Whether all results that have shown up are relevant.

Recall: Has to do with completeness. Whether all valid/ relevant results have shown up.

Needs detailing..

Pentaho 5.0 Community Edition Released

The stable build of Pentaho Community Edition (CE) 5.0 has been released. Many many new features have made it to this build. Particularly keen to try out the enhancements to web services deployment via Carte. More details can be found in the Pentaho 5.0 release notes. 

Pentaho Clusters

Pentaho provides the option to scale out Kettle Transformations via Pentaho Clusters. It is fairly straightforward to set up a Pentaho cluster and elastic/dynamic clusters. The 1-2-3 of what needs to be done is:

1. Start the Carte Instances
There are two kinds of instances - Masters & Slaves. At least one instance must act as the dedicated Master which takes on the responsibility of management/ distribution of transformations/ steps to slaves, fail-over/ restart and communicating with the slaves.

The Carte instances need a config file with details about the Master's port, IP/ Hostname etc. For sample config files take a look at the pwd folder in your default Pentaho installation (/data-integration/pwd).

E.g. With defaults, a cluster can be started on localhost with:


2. Set up Cluster & Server Information using Spoon (GUI)
Switch to the View tab, next to the Design tab in the left hand panel of the Spoon GUI.
Click on 'Slave Servers' to add new Slave servers (host, port, name, etc.). Make sure to check the 'is_the_master' checkbox for the Master server.

Next click on the 'Kettle Cluster Schemas' and use 'Select Slave servers' to choose the slave servers. For  the ability to dynamically add/ remove slave servers, also select the 'Dynamic Cluster' checkbox.

3. Mark Transformation Steps to Execute in Cluster Mode
Right click on the step which needs to be run in the cluster mode, select Clustering & then select the cluster schema. You will now see a symbol next to the step (CxN) indicating that the step is to be executed in a clustered mode.

The cluster settings will be similar to what you see in the left panel in the image. You can also see a transformation, with two steps (Random & Replace in String) being run in a clustered mode in the right panel in the image below.

Shanon Entropy and Information Gain

Shanon's Information Gain/ Entropy theory gets applied a lot in areas such as data encoding, compression and networking. Entropy, as defined by Shanon, is a measure of the unpredictability of a given message. The higher the entropy the more unpredictable the content of the message is to a receiver.

Correspondingly, a high Entropy message is also high on Information Content. On receiving a high Entropy/ high Information Content laden message, the receiver has a high Information Gain.

On the other hand, when the receiver already knows the contents (or of a certain bias) of the message, the Information Content of the message is low. On receiving such a message the receiver has less Information Gain. Effectively once the uncertainty about the content of the message has reduced, the Entropy of the message has also dropped and the Information Gain from receiving such a message has gone down. The reasoning this far is quite intuitive.

The Entropy (& unpredictability) is the highest for a fair coin (example 1.a) and decreases for a biased coin (examples 1.b & 1.c). Due to the bias the receiver is able to predict the outcome (favouring the known bias) in the later case resulting in a lower Entropy.

The observation from the (2-outcomes) coin toss case generalizes to the N-outcomes case, and the Entropy is found to be highest when all N-outcomes are equally likely (fair).

Be Hands On

For as long as you can. Think specialists (surgeons, pilots, etc.) who get better clocking in more hours with/into their art - doing, practicing, persevering. 

General Availability (GA) for Hadoop 2.x

The Hadoop 2.x GA is nothing less that a big leap forward. Most of the features released such as YARN - a pluggable resource management framework, Name Node HA, HDFS Federation and so on were long awaited. As per the official mail to the community, this release includes:

"To recap, this release has a number of significant highlights compared to Hadoop 1.x:
        • YARN - A general purpose resource management system for Hadoop to allow MapReduce and other other data processing frameworks and services
        • High Availability for HDFS
        • HDFS Federation
        • HDFS Snapshots
        • NFSv3 access to data in HDFS
        • Support for running Hadoop on Microsoft Windows
        • Binary Compatibility for MapReduce applications built on hadoop-1.x
        • Substantial amount of integration testing with rest of projects in the ecosystem

 Please see the Hadoop 2.2.0 Release Notes for details."

Also as per the official email to the community, users are encouraged to move forward to the 2.x branch which is more stable & backward compatible.

Need Support to Lift with Confidence

Brace up terminologies coming your way...

Support: A measure of the prevalence of an event x in a given set of N data points. Support is effectively a first level indicator of something occurring frequent enough (say greater than 10% of the times) to be of interest.

In the case of two correlated events x & y,

Confidence: A measure of predictability of two events occurring together. Once confidence is above a certain threshold (say 70%), it means the two events show up together often enough to be used for rules/ decision making, etc.

Lift: A measure of the power of association between two events. For an event y that has occurred, how much more likely is event y to occur once it is known that event x has occurred

False Negative, False Positive and the Paradox

First a bit about the terms False Positive & False Negative. There terms are associated with the nature of error in the results churned out by a system trying to answer an unknown problem, based on a (limited) set of given/ input data points. After analysing the data, the system is expected to come up with a Yes (it is Positive) or a No (it is Negative) type answer. There is invariably some error in the answer due to noisy data, wrong assumptions, calculation mistakes, unanticipated cases, mechanical errors, surges, etc.

A False Positive is when the system says the answer is Positive, but the answer is actually wrong. An example would be a sensitive car's burglar alarm system that starts to beep due to heavy lightning & thunder on a rainy day. The alarm at this stage is indicating a positive hit (i.e. a burglary), which is not really happening.

On the other hand, a False Negative is when the system answers in a Negative, where the answer should have been a Positive. False negatives happen often with first level medical tests and scans which are unable to detect the cause of pain or discomfort. The test report of "Nothing Abnormal Detected" at this stage is often a False Negative, as revealed by more detailed tests performed later.

The False Positive Paradox is an interesting phenomenon where the likelihood of a False Positive shoots up significantly (& sometimes beyond the actual positive) when the actual rate of occurrence of a condition within a given sample group is very low. The results are thanks to basic likelihood calculations as shown below.

Let's say in a group of size 1,000,000 (1 Mn.), 10% are doctors. Let's say there's a system wherein you feed in a person's Unique ID (UID) and it tells you if the person is a doctor or not. The system has a 0.01% chance of incorrectly reporting a person who is not a doctor to be a doctor (a False Positive).

Now, let's work out our confidence levels of the results given out by the system.


On the other hand if just 0.01% of people in the group are actually doctors (while the rest of the info. remains same) the confidence level works out to be quite different.


This clearly shows that the likelihood of the answer being a False Positive has shot up from much under 1% to as much as 50%, when the occurrence of a condition (number of doctors) within a given population dropped from 10%  (i.e. 100,000) to a low value of 0.1% (i.e. 1,000).

Transparently

While doing software development you might hear of change being introduced "transparently". What does this mean?

Transparency in this context is similar to how a looking glass is transparent. One can barely make out that it exists. Think of a biker who pulls down the glass visor of his helmet when troubled by wind blowing into his eyes. His sight of the road & beyond continue to function without his noticing the transparent visor layer in-between.

Similarly, when a change in introduced transparently on the server side, it means the dependent/ client side applications needn't be told/ made aware of this change on the server side. The old interfaces continue to work as is, communication protocols remain the same, and so on.

The above kind of transparency is different from the transparency of a "transparent person" or a "transparent deal" or a "white box system", where the internals (like thoughts, implementation, ideas, details, etc.) are visible.

Identity Function and Idempotence

Internals Of Solr/ Lucene Document Scoring

This post is in continuation a discussion on the solr community about the efficiency of Solr/ Lucene scoring algorithm.

The search algorithm given here can be summarized to:

- Query query =  Build query using user's search terms.
- Collector collector = Typically the TopScoreDocCollector
- Searcher searcher = new IndexSearcher(indexReader);
- searcher.search(query, collector);
- Weight weight = query.weight(searcher);
- Scorer scorer = weight.scorer(indexReader); // Typically BooleanScorer2
- scorer.score() => ConjunctionScorer (on every sub-scorer) in a leap frog/ skip ahead mechanism.

Algo needs improvement!

The AND query shows a leap frog/ skip ahead ahead pattern implemented in the BooleanScorer2 (ConjunctionScorer) level.

For example with the query, q=A AND B, where A & B match doc. id's
A -> 1,3,5,7,11,15,17
B -> 2, 6

- Scorer starts with the min. of each, i.e. A -> 1  & B -> 2, & current highest doc id set to 2

- In the next few iterations:
A is advanced past the current highest value to 3 & current highest updated to 3.
B advanced past current highest 3 to 6 & current highest set to 6.
A advanced past 6 to 7 & current highest set to 7.
B has no more docs & this breaks out, without any match.

On the other hand if the two had converged/ agreed on a particular doc id, that doc would be scored & collected (added to a min-heap of scores).

Can you fly that thing? Not yet...

Helps if you bring this spirit to the team.

Update Apt Repositories Location for Old Ubuntu Versions

When working with an old versions of Ubuntu (11.04, 10.04, etc.), the biggest handicap is the lack of a functional package manager such as apt or synaptic. The reason why the package managers stop working is that at end of support/ licence for an old version of Ubuntu, the team behind Ubuntu archive the repositories.

At this point as an user you are supposed to Upgrade (the recommended practice) to a more recent version. There are normally enough advance notices and alerts sent out by Ubuntu's Update Manager for the same. If however, you have a compelling reason to stick on to your current version, then here's a   way to update your repositories' sources list to be able to install and use old software that is present in the archival repository. This is based on the recommendation made on this forum discussion.

Resume Large Downloads in Mozilla Firefox

When downloading a large file via Firefox over a slow internet connection you might get disconnected in between and end up with a partially downloaded file (with a .part file extension). Here's a little trick to Resume the download after reconnecting, on wards, from where the download had stopped previously.

1. Open up the Firefox Downloads window (Tools > Downloads OR use shortcut Ctrl+ Shift + Y). Not sure if this works with the recent versions of Firefox.

2. Click on the Resume button next to the file you were downloading/ got downloaded partially. If this works then great, nothing else to do.

3. On the other hand, if step 2 didn't work, then click on the Retry button. This will result in the download to start off all over again from the very beginning. Let it start and go over to step 4.

4. Once a few bytes of the file has been downloaded & the progress meter on the Download window indicates that the new download has started (might also give an estimate of time left), click on the Pause button next to the download.

5. Now go to your Downloads folder (where Firefox was downloading the file). Rename the first file that was partially downloaded (having the file name extension as .part) to the new file that just started downloading in step 4.

6. Go back to the Firefox Downloads window and click the Resume/ Start/ Restart button next to the download process that was Paused in step 4.

That's it. The download should resume from the point where the initial partially downloaded (.part) file had stopped.

Trees and Graphs

Useful things to know about the trees and graphs based data structures:

These:

• Binary Trees Vs. Binary Search Trees
• 2,4 and Red Black
• AVL
• Tries
• Heaps
• B & B+ Trees

& these:

• BFS, DFS
• Sorting - Quick, Merge, Radix, Timsort
• Kruskal's & Prim's algorithm for Minimum Spanning Trees
• Morris Traversal, without extra space or recursion, using Threaded Binary Trees
• Djikstra's algorithm for shortest path
• Topological sorting

That horses for courses is applicable:

The big-O deal:

With Java, well tested implementations are mostly available:

• TreeSet
• TreeMap
• LinkedHashMap
• ConcurrentSkipListMap
• PriorityQueue

Zookeeper for Synchronizing Across Distributed Systems

Find it all on Apache Curator.

Solr Analyzers Basics

Solr offers several Analyzers to pre-process document fields being indexed and searched. As part of modelling the schema one needs to make an informed choice for the specific chain of Analyzers to be applied to every field (fieldType) defined in the schema.xml.

To start off one needs to understand that different kinds of Analyzers and their purpose:

1. Char Filters (or CharacterFilterFactories)
   
• Always applied first, i.e. before Tokenizers
• Operates at the Character level (of the field values)
• Zero or More Char Filters can be chained together. Get applied as per the sequence in schema.xml
2. Tokenizers (or TokenizerFactories)
   
• Converts stream of Characters into a series of Tokens
• Only One Tokenizer can be there in each Analyzer chain
3. Token Filters (or TokenFilterFactories)
• Always applied last, i.e. after Tokenizers
• Operates at the Tokens level generated by the Tokenizers
• Zero or More Token Filters can be chained together. Get applied as per the sequence in schema.xml

To take an example, let's say we have a field title with the value (V1) "Mr. James <b>Bond</b> MI007". Now we run it through the following:

1. Character FilterFactory (One): HTMLStripCharFilterFactory (CF1)

(Output: "Mr. James Bond MI007")

2. Tokenizer (One): StandardTokenizerFactory (T)

(Output: Tokens: [ALPHANUM: "Mr.", ALPHANUM:"James", ALPHANUM:"Bond", ALNUM:"MI007"])

3. TokenFilters (Two): WordDelimiterFilterFactory (TF1) & LowerCaseFilterFactory (TF2) 

• Mr. => WordDelim => Lowercase => mr.
• James => WordDelim => Lowercase => james
• Bond => WordDelim => Lowercase => bond
• MI007 => WordDelim => [MI, 007] => Lowercase => mi, 007

Finally the output text actually indexed: "mr. james bond mi 007"

There are several other options and many more Analyzers that one could.  Among them the different PatternReplace Analyzers, EdgeNGram and the simple WhiteSpaceFilterFactory are the more popular ones.  Finally, if none of the standard ones are adequate for a specific use case then there is also the option of writing a custom analyzer.

Solution for making Long GET Request to Solr via SolrNet

Solr has REST api's available for performing various searches on indexed documents. The client generally issues GET requests to Solr with different parameters (fields, row, facet, etc.) set. Since there typically are size/ query length limitations on GET requests (imposed by container, OS, etc.), Solr allows the same queries to be issued to the Solr RequestHandlers as POST request as well.

We ran into one such issue with long GET request to Solr from SolrNet and did a few changes to solve the same.

Solr Side Changes:
First up, we increased the headerBufferSize of the application server as explained on SO here and increased the maxBooleanClauses parameter in solrconfig.xml. This allowed Solr side to start responding to much longer GET requests. The problem however wasn't solved. The client side was a dot net application running within IIS having additional length limitations imposed by Windows OS & the dot net framework.

SolrNet Side Changes:
In round two, we went for a better fix and switched over to a POST requests in place of long GET requests. The solution is largely the same as mentioned on the SolrNet group here & here. The difference being to switch over to a POST request from within the Get() method of the SolrConnection.cs class, when the request string is longer than a configurable threshold value.


Update: PostSolrConnection.cs class has made it to the head branch of SolrNet.  

Redmine Project Management Tool

In trying to find an Open source Agile project management tool, somewhat of an alternative to Rally, chanced upon Redmine. The initial feel of the tool has been good so far.

Needed somewhat of an integrated tool that would allow various teams to collaborate. Redmine does well on this count as it has a task tracker, bug tracker, knowledge repository (file/ document management and wiki), all rolled in to one.

Additionally, we have been able to migrate our bugs and user accounts from Bugzilla, to get off the ground quick. Now it is about letting rubber hit the road, and having the teams to start working with Redmine. 

SolrNet Separate Highlighting Query - hl.q

Solr allows highlighting of matched sections in field values.  There are several parameters that can be set by the caller to adjust the highlighting behaviour.

SolrNet, a library to connect to Solr from dot net applications, also has HighlightingParameters exposed in SolrNet core library. However, not all/ a very small subset of parameters are currently exposed.

Recently needed to use the hl.q query, to issues a separate/ more specific highlighting query to Solr. The work around was to make use of the ExtraParams option, from the base CommonQueryOptions class.

The same approach could be used for any of the other parameters not exposed by SolrNet, such ais hl.BoundaryScanner, per field highlighting, maxScan, etc., essentially all the 3.5x onward features mentioned on the Solr Highlighting wiki.

Php Script To Display Process, Vmstat, Disk Usage, Syslog Of A Linux Server Via A Browser

A Php script that executes some standard shell programs for monitoring resource utilization & processes on a given Linux box. The script directs the output to a web-browser.

Apache web-server should be installed on the server.  To run copy the script to the  DocumentRoot (/var/www/html). Appropriate execute rights (-rw-x) need to be given to the apache user (which runs this script, but is not the owner) to execute this Php file & to be able to access /var/log/syslog.

Save this file as: showHealth.php in the /var/www/html folder:

Linux/Unix Shell Function For Date Addition and Subtraction

Here is a small shell script to do date addition and subtraction. This works on the bash shell with GNU Date.

Upload to Amazon S3 Bucket via Signed Url with Server Side Encryption

Continuing further from my previous post on upload & download from Amazon S3 bucket via signed url's, here is how to enable Server Side Encryption (SES) with the file being uploaded to S3.

Add a x-amz-server-side-encryption request parameter with the GeneratePresignedUrlRequest before getting the signed url:

Upload and Download from Amazon AWS S3 Bucket via Signed Url

While the code snippets are using the Java AWS SDKs, principally these will work with the other SDKs as well.


1. Get hold of FederatedCredentials using your AWS credentials:

Pass in proper access Policy settings for the FederatedCredentials on the S3 Bucket and/ or Item.

E.g.



For Download you could additionally set up ResponseHeaderOverrides for withContentDisposition, ContentType, etc.


2. Get BasicSessionCredentials using the Federated Credentials


3. Generate GeneratePresignedUrlRequest

4. Finally, generate a pre-signed url via the S3Client object:


5. To test this:
- Download:
Get the url.toString() & hit it from a browser


- Upload:

Uploading Large Files In Chunks To Amazon S3

A collection of best practices based on my experience building a scaled out solution for the server side file upload handler.

1. Authentication/ Authorization

2. Chunking

3. Stateless upload & Session

4. Shared memory for post file operations

5. Retries & Failover

6. Bulk operations

To be completed.. 

Autovue Jump To Page

Autovue is a browser based document viewing & markup application.  
To open up a specific page of a document in the viewer simply set up an ONINIT javascript call back method via the applet param.



You can get more info on this from the Advanced Scripting Functionality section of the InstallConfigGuideCS of Oracle Autovue. 

Atomic Updates via SolrNet

As of today the SolrNet api doesn't offer atomic updates to be issued to a running Solr server. While the Solrnet api is supposed to offer this feature sometime in the future, the following alternative can be used in the interim.

1. Build a custom atomic update XML message:


(See: http://wiki.apache.org/solr/UpdateXmlMessages for more details)

2. Get hold of the connection object (via ServiceLocator):


3. Issue a call to Solr via the connection object:


Will be adding sample code snippets soon..

Solr Cell, Tika And Pages

With Solr Cell, aka Tika, you get the power to index content from within  a wide set of digital files such as Pdfs, Office, Text, etc.

Tika however doesn't naturally offer any demarcations for page boundaries. So you can search for content matches from a file, but not for specific pages from within these files.

Among several different ways to solve this problem, one way could be to index each page of the file as a separate document in Solr and do a field collapsing/ result grouping on the search results by a common file identifier shared by all pages of the file.

Since there could be performance overheads with result grouping, another way is to index the combined file as one solr document (of type Combined) & each page as a separate solr document (of type Page) with a common file identifier. The search can then be performed initially against the combined document (type:combined AND text:abc) to identify files that match & then against the corresponding page type document (type:page AND file-id:123 AND text:abc) to identify pages.

Mocking AWS ELB Behaviour Locally For Testing

Once hosted out of Amazon, you make use of the AWS Elastic Load Balancer (ELB) for balancing load across your EC2's within or acroos Availability Zones (AZ). Since code gets developed and tested locally (outside of Amazon), at times you might want to test load balancer scenarios before deploying to production. Here's one way to mock up the load balancer behaviour for local testing.

Use Apache (you could very well use something like Nginx instead) in a reverse proxy, load balancer set up via mod_proxy & mod_proxy_balancer.  Fairly simple for anyone with slight experience with configuring Apache. We used Apache as a load balancer front-end to IIS on local, exactly the way ELB would load balance in front of production IIS.

Additionally, since ELB was also an SSL end point for our production servers, we set up Apache to be the SSL end point (via mod_ssl) on local. Apache was configured to listen on port 443 (using a self-signed certificate), and would forward all traffic from port 443 to backend IIS on port 80.

Once we had that set-up going, we were quickly able to reproduce an issue with application generated Secure cookies not getting set properly across client request/ response. Once we had the fix on the local (which was to set the flag on the cookies in the request, not response) the same worked flawlessly on the AWS as well.

Headless Java Monster

You know you are up against the same fellow if you start seeing the

java.awt.HeadlessException, typically running off a virtual server, or in the rare  case of a dedicated server without a monitor (aka head).

The solution is simple. First shut down the application, tomcat, etc. that got the exception.

1. Install the X display manager. 

On a Ubuntu on the other hand, you could install the Xvfb package (via apt, synaptic, etc.)

2. Start X:

3. Export display:
With those done, now you should have entered the simpler "No X11 DISPLAY variable" zone. Simply export the display variable to fix this.

(In the Ubuntu case above you have to export DISPLAY=:1)

4. Allow all users to connect/ use this Display variable:

Now restart the application, tomcat, etc. that you were trying to run initially & it should work. Hope nothing headless ever troubles no man!

C# - A home away from home

For someone with years of experience in Java, the stint to code in C# seemed like a cake walk. The large scale port of popular Java frameworks into dot net such as NHibernate, Spring.net, NUnit, etc., make life all that much easier for anyone starting to bridge the gap.

There are however some bits that trouble us Java natives no end. Particularly anything & everything to with the web.config file. This one file has enough traps in it to confuse the hell out of any sane minded developer. The file has hints for the IDE (Visual studio), the framework (Asp.net), the web server (IIS), & everyone else connected to the runtime.

The other bit that seems bothersome is how dependencies, packages & Dll's get referenced. Particularly with frequently changing Dll's, & the tools optimized for caching, it gets difficult to know what version is really loaded & running.

Anyway, once you get past these issues, the ride ahead is through familiar grounds.

Using Pentaho Kettle to Index Data in Solr

Pentaho Kettle is a fine open source ETL tool written in Java. There are several implementations, hooks and plugins available off the shelf for performing various Extract (E), Transform (T), Load (L) processes on data from a source location to a destination location.

Solr, on the other hand, is a rich and powerful production grade search engine written on top of Lucene. So how would it be to get the two to function in tandem? To use Kettle to load data into Solr for indexing purpose.

The data load phase for indexing in Solr is very similar to an ETL process. The data is sourced (Extract) from a relational Database (MySql, Postgre, etc.). This data is denormalized and transformed to a Solr compatible document (Transform). Finally the transformed data is streamed to Solr for indexing (Load). Kettle excels in performing each of these steps!

A Kettle ETL job to load data into Solr for indexing, is a good alternative to using Solr's very own Data Import Handler (DIH). Since DIH typically runs off the same Solr setup (with a few common dependencies) so there's some intermixing of concerns with such a set-up,  between what Solr is good at (search & indexing) versus what the DIH is built to do (import documents). The DIH also competes for resources (CPU, IO) with Solr. Ketttle has no such drawbacks and can be run off a different set of physical boxes.

There are additional benefits of using Kettle such as availability of stable implementations for working across data sources, querying, bulk load, setting up of staged workflows with configurable queues & worker threads. Also Kettle's exception handling, retry mechanism, REST/ WS client, JSON serializer, custom Java code extension, and several handy transformation capabilities, all add up in its favour.

On the cons, given that the call to Solr would be via standard REST client from Kettle, the set-up would not be Solr Cloud or Zookeeper (ZK) aware to be able to do any smart routing of documents. One option to solve this could be to use the Custom Java Code step in Kettle and delegate the call to Solr via the SolrJ's CloudSolrServer client (which is Solr Cloud/ ZK aware).