Showing posts with label Storm. Show all posts
Showing posts with label Storm. Show all posts

Tuesday, August 1, 2017

On Storm

Storm Fundamentals
  • http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/
  • https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_storm-component-guide/content/ch_storm-topology-tuning.html
  • http://grokbase.com/t/gg/storm-user/12c56ep4dk/calculating-the-capacity-of-a-bolt
  • http://storm.apache.org/releases/2.0.0-SNAPSHOT/Logs.html
  • https://stackoverflow.com/questions/35864128/how-to-set-storm-workers-jvm-max-heap-size
  • https://stackoverflow.com/questions/20914631/configuration-of-workers-in-a-storm-cluster
  • http://storm.apache.org/releases/1.1.0/Concepts.html

Metrics, Debugging, Monitoring, Logging

  • https://community.hortonworks.com/articles/36151/debugging-an-apache-storm-topology.html
  • http://storm.apache.org/releases/1.0.3/Metrics.html
  • https://www.opsclarity.com/monitoring-troubleshooting-apache-storm-opsclarity/
  • http://storm.apache.org/releases/2.0.0-SNAPSHOT/Logs.html
  • https://etl.svbtle.com/visualizing-metrics-in-storm-using-statsdgraphite
  • http://www.brianhsieh.com/2014/06/nagios-for-monitor-kafka.html
  • https://etl.svbtle.com/visualizing-metrics-in-storm-using-statsdgraphite
  • https://dzone.com/articles/monitoring-and-troubleshooting-apache-storm-with-o
  • https://community.hortonworks.com/articles/36151/debugging-an-apache-storm-topology.html (Logs)

Backpressure, Buffer, etc

  • http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/
  • https://stackoverflow.com/questions/44557915/backpressure-in-storm
  • http://jobs.one2team.com/apache-storms/
  • https://issues.apache.org/jira/browse/STORM-1949 (Issues with Backpressure implementation)
  • http://storm.apache.org/releases/0.10.0/Configuration.html (MaxSpoutPending + acking seems like the only option for now)


Monday, August 10, 2015

Fast Streaming Solution

High level view:


Web API          -> Kafka                     -> Storm (Streaming)

                    -> Hadoop/ HDFS    -> MR/ Hive (Batch)

 

Specifics TBD..