Monday, September 30, 2024

Restore Joomla 4 Manually

This post has info on manually restoring a Joomla 4.3.4 set-up across two servers. While both are Linux systems the configuration differ slightly including the OS, Php , DB, etc. Various issues were faced & overcome in doing the restoration.

Background info:

- Source:
Ubuntu 22.04, Php 8.2, Joomla 4.3.4, Apache, Maria DB, Addon Plugins (AddToAny, LazyDb, Komento, SexyPolling)

- Destination:
Ubuntu 20.04, Php 7.4, Joomla 4.3.4, Apache, MySql 8.0

- Latest DB dump & htdocs folder (including all files, modules, plugins, media, images etc.) from Source was transferred to Destination server via Ftp before hand.

Steps:

1) DB Import

    1.1) Create user, db, grant all permission to user. 

    1.2) Import data to the created db from the latest DB dump of the source.
 
        1.2.1) ERROR 1366 (HY000) at line 2273: Incorrect integer value: '' for column 'checked_out' at row 1. Solution is to set NO_ENGINE_SUBSTITUTION & then import:

            SET @@GLOBAL.sql_mode= 'NO_ENGINE_SUBSTITUTION';

    1.3) ERROR 1101 (42000) at line 10692: BLOB, TEXT, GEOMETRY or JSON column 'country' can't have a default value

        - Using the solution found online & the DB dump sql import script was changed to set DEFAULT values for the problematic text columns country, city, etc
    

         // Modify the sexypolling plugin CREATE TABLE script:
            CREATE TABLE `#_sexy_votes` (
              `id_vote` int(10) unsigned NOT NULL AUTO_INCREMENT,
                  ....
              `country` text NOT NULL DEFAULT (_utf8mb4'Unknown'),
              `city` text NOT NULL DEFAULT (_utf8mb4'Unknown'),
              `region` text NOT NULL DEFAULT (_utf8mb4'Unknown'),
              `countrycode` text NOT NULL DEFAULT (_utf8mb4'Unknown'),

              PRIMARY KEY (`id_vote`),
                  .....

            ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci;

2) Download Joomla_4.3.4-Stable-Full_Package.zip from joomla.org
    2.1) Unzip Joomla_4.3.4-Stable-Full_Package.zip to /var/www/html & rename folder to <site_name>

    2.2) Set up site configuration.php (/var/www/html/<site_name>/configuration.php)
        - Add db, username, password
        - Add tmp_path & log_path in

        public $log_path = '/var/www/html/<site_name>/administrator/logs';
          public $tmp_path = '/var/www/html/<site_name>/tmp';


3) Restore Joomla modules, plugins, languages, etc from file system Ftp backup location of Source.

4) Additional system settings on Destination


    4.1) Add missing Php modules: "Call to undefined function" error

        4.1.1) simplexml_load_file()
            sudo apt-get install php7.4-xml

        4.1.2) "IntlTimeZone" module missing
            sudo apt-get install php7.4-intl

    4.2) Increase Php upload limit (/etc/php/7.4/apache2/php.ini)

          post_max_size = 38M
            upload_max_filesize = 32M


    4.3) Restart apache
          sudo systemctl reload apache2
 

5) Recovering from J4 Red Error Page of death

    5.1) Redirection to installation/index.php:

    - With an error "500 - Whoops, looks like something went wrong".
    - Needed to delete the installation folder, to stop the redirection.

    5.2) Next, 404 Component not found error on the home page: 
 

    ---
        404 **Component not found.**

        Call stack
        #     Function     Location
        1     ()     JROOT/libraries/src/Component/ComponentHelper.php:296
        2     Joomla\CMS\Component\ComponentHelper::renderComponent()     JROOT/libraries/src/Application/SiteApplication.php:210
        3     Joomla\CMS\Application\SiteApplication->dispatch()     JROOT/libraries/src/Application/SiteApplication.php:251
        4     Joomla\CMS\Application\SiteApplication->doExecute()     JROOT/libraries/src/Application/CMSApplication.php:293
        5     Joomla\CMS\Application\CMSApplication->execute()     JROOT/includes/app.php:61
        6     require_once()     JROOT/index.php:32
    ---
 

    5.3) Checked DB connections using a custom php script:

No issues connecting to DB with username/ password!

    5.4) Enable Debugging/ Logging:
        

        5.4.1) Logging in php.ini (/etc/php/7.4/apache2/php.ini)
            ----Turn on logging-----
               display_errors = On
               html_errors = On
               display_startup_errors = On
               log_errors = On
               error_log = /var/log/apache2/php_errors.log


        5.4.2) Logging in configuration.php (/var/www/html/<site_name>/configuration.php)
            // Change to true from false
                public $debug = true;
                public $debug_lang = true;

                // Change to 'maximum' from 'default'
                public $error_reporting = 'maximum';
    
                // Change to 1 from 0
                public $log_everything = 1;


    With those J! Info started showing up in the browser along with the error stack trace & queries.

    5.5) Root cause analysis

        5.5.1) Checked the specific php libraries:
            libraries/src/Component/ComponentHelper.php:296
            libraries/src/Application/SiteApplication.php:210, etc..

    - Using var_dump($component), on SiteApplication.php:210 found: 

           $component = NULL

        - The same '$component = "com_content"' on the home page of a default Joomla application (unzip Joomla 4.3.4 zip & install & check value on Joomla home page).

        - Test with hard coded $component = "com_content" in libraries/src/Application/SiteApplication.php:210

                if(empty($component)){
                     $component = "com_content";
         
       }


        - With this 404 was gone & a broken site home page came up with a few Category links listed out

        - Clicking on Category link was showing that "No Article linked to Category", despite there being several Articles imported from source db dump.

        5.5.2) Localizing issue with Content/ Article loading:
        - Hit the direct Article url:
            http://<site_name>/index.php?option=com_content&view=article&id=<article_id>

        - This gave another error"404 Article not found", though the specific <article_id> was present in the database.

        - J! Info provided the corresponding php file and db query used to fetch article by id which was giving no result

    5.5.3) Issue with imports of all "datetime DEFAULT NULL" fields

    - On exploring the query further, it was seen to have checks for publish_up & publish_down dates. These needed to be either NULL or set to a date earlier (/later) than date NOW for publish_up (/publish_down).

    - In the "#_content" table publish_up & publish_down dates values were showing as "0000-00-00 00:00:00" (i.e. were imported as 0) in place of NULL.
This was causing all records being filtered out.

    - It also meant that wherever the "datetime default NULL" fields were imported the same issue was happening. 

        - A check revealed 30 other J! tables with the same issue.
 
        - Prepared a script to update each of these datetime fields to NULL in the 30 tables.
            UPDATE `#_content` SET `checked_out_time` = NULL, `publish_up` = NULL, `publish_down` = NULL;

            UPDATE `#_categories` SET `checked_out_time` = NULL;
                                                                     .... 
for all the affected tables!

With that the issue was resolved & site home page became functional!

Saturday, September 28, 2024

Sexy Polling Reloaded Extension (Plugin) on J4 with MySql 8

For anyone installing the plugin SexyPolling 4.1.7 on a Joomla 4.3.4 with a MySql 8.0 db on an Ubuntu system there may be issues with default value for Text fields. More specifically error in setting default value for country, city, etc TEXT fields:

    "BLOB, TEXT, GEOMETRY or JSON column 'country' can't have a default value"

1) There is a solution for MySql 8.0 to set DEFAULT values for TEXT fields. The CREATE statement for table `#_sexy_votes` needs to be changed to:

         // Modify the sexypolling plugin CREATE TABLE script:
            CREATE TABLE `#_sexy_votes` (
              `id_vote` int(10) unsigned NOT NULL AUTO_INCREMENT,
                  ....
              `country` text NOT NULL DEFAULT (_utf8mb4'Unknown'),
              `city` text NOT NULL DEFAULT (_utf8mb4'Unknown'),
              `region` text NOT NULL DEFAULT (_utf8mb4'Unknown'),
              `countrycode` text NOT NULL DEFAULT (_utf8mb4'Unknown'),
              PRIMARY KEY (`id_vote`),
                  .....
            ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci;


Further, in order for the CREATE table changes to take effect & not get dropped/ altered, a few of the plugin's installer scripts needs to be modified. The installation then needs to be done from a folder (or a modified zip file) in which the modified plugin installer scripts are present (instead of the downloaded joomla_plugin_sexypolling_reloaded_v4.1.7.zip file), as explained next.

2) Installation of plugin from /tmp folder 

2.1) Unzip the downloaded joomla_plugin_sexypolling_reloaded_v4.1.7.zip file to the site /tmp folder as mentioned in the site configuration.php file (e.g. /var/www/html/<site_name>/tmp). Give the folder proper read/write permissions.

2.2) Change CREATE TABLE `#_sexy_votes` command  in /tmp/com_sexypolling/admin/install/sql/install.sql:

Set the DEFAULT value for country, city, region, countrycode to  "DEFAULT (_utf8mb4'Unknown')" as mentioned above.
 
2.3) Remove ALTER TABLE `#_sexy_votes` command from /tmp/com_sexypolling/scriptfile.php. 

     Put an invalid condition check on line 235 of the script to stop ALTER TABLE for `#_sexy_votes` to run:

         $alterSexyVotes=false;

         if($alterSexyVotes && is_array($columns_titles)) { 

            ...

2.4) Finally, install:

  • Option A: Install from modified com_sexypolling folder: Install the extension from the folder /tmp/com_sexypolling from the J! "Admin" > "System" > "Install Extensions" > "Install from Folder" page.
  • Option B: Zip & Install modified com_sexypolling folder: Another option could be to manually zip the folder with the modifications (/tmp/com_sexypolling) & install using this zip file from the standard J! "Admin" > "System" > "Install Extensions" > "Upload Package File" page.

 

Monday, August 19, 2024

Pygradle for Python-3

Gradle, the build workhorse from the Java ecosystem, extends its support to Python through Pygradle. A recent attempt to build a Python-3.x project using Pygradle though did't work as expected. 

The delta between the supported Python-2.x vs Python-3.x is hard to reconcile with many issues like:

  • Need for a specific, old version of Java (ver.8), Gradle (ver. 5.0), etc
  • Dependencies on old versions of Python modules without backwards compatibility
    • Hard to figure out which exact version will work
    • A rule of thumb is to pick the highest version dependency module around some cut-off year like 2018/19, post which they don't seem to build
  • Downloading of the correct dependencies & creating ivy files
    • Includes identifying the right version, name, dependencies-within-dependencies (that no longer work on Python-3.x), etc.
  • Using a local file system based repo to download & build modules & ivy files

With some effort though, have been able to complete a successful build on a Python-3.8 on an Ubuntu-20.04 with Java-8 & Gradle-5.0. More details are available on the pygradle_python3_example repo. Hope this helps!

Monday, July 22, 2024

Cloudera - Streaming Data Platform

Cloudera has a significantly mature streaming offering on their Data Platform. Data from varied sources such as rich media, text, chat, message queues, etc is brought in to their unified DataFlow platform using Nifi or other ETL/ ELT. After processing these can be directed to one or more of the Op./ App DB, Data Lake (Iceberg), Vector DB post embedding (for AI/ ML), etc.

Streaming in AI/ ML apps help to provide a real-time context that can be leveraged by the apps. Things like feedback mechanism, grounding of outputs, avoiding hallucinations, model evolution, etc all of them require real-time data to be available. So with a better faster data, MLOPs platform Cloudera is looking to improve the quality of the ML apps itself running on them.

Cloudera has also made it easy to get stared with ML with their cloud based Accelarators (AMP). AMPs have support for not just Cloudera built modules, but even those from others like Pinecode, AWS, Hugging Face, etc & the ML community. Apps for Chats, Text summarization, Image analysis, Time series, LLMs, etc are available for use off the shelf. As always, Cloudera continues to offer all deployment options like on-premise, cloud & hybrid as per customer's needs.

 

Monday, April 1, 2024

Gurobi Optimizer

Gurobi stack consists of various modules in Python (Gurobipy) & other languages for solving Optimization problems. Think of scheduling, routing, cost minimization, profit maximization, flow decision, OR, assignment and so on. They have all the classical Linear Programming, ILP, Greedy, Constrained Optimization type algos properly implemented & ready for use at scale.

Off late they are fusing Mathematical Optimizations with AI to yield much better modules. Their stack includes Neural Nets, DNN, Differential Programming, Simulators, Reinforcement Learning and all the other tools required to fuse ML & Optimization. They also have hardware & cloud offerings now. These can be good places to start of by uploading modules that need to be optimized on demand. The output results can be integrated within applications/ workflows.   

Monday, March 25, 2024

VR Movies

Watching movies on VR headset was an experience like none other. Thanks to the organizers from Goethe Institute & IHC for hosting such a unique event for the viewers.

Immersive movies & 360 filmmaking is a rather new development taking place in the world of cinema. International events like the VRHAM Film Festival is exclusively being organized since 2018 for VR movies makers. All the biggies from the tech world are in on it with VR headsets catering to the growing demands. So the action is hotting up on the AR/ VR front.

Going back the movies, there were about ten of them on display. These were short ones of about 15 min each curated by U. Schauth.  I got the opportunity to watch:

Each movie was remarkable. Distinct stories, providing an immersive experience to our connected pasts. Highly recommend the experience to all.


Sunday, March 10, 2024

Gaming World

Recently got an opportunity to attend the India Gaming Summit 2024 in person. This was truly an insightful first peek into the world of the gamers!

All kinds of gaming professions have been gaining traction over the last few years. On the one end of the spectrum are the elite ESport Atheletes who compete on national & international forums. On the other there are Casual Gamers, playing games on their mobiles & computers for fun, entertainment, and as a recreational pass time. 

On a slightly different tangent are the gamers playing Real Money Games (RMG). They play games (like poker, etc) with actual monetary bets & rewards (losses). There is a wide age profile mix among all the different kinds of gamers. From teenagers to senior citizens to home makers to professionals it seems the whole world is busy playing games, not just in their free time. Though preferences for different types of games vary across demographics.

Those considerations and more are the lookout of the other kind of gaming professionals - the game developers. These are the tech & biz folks building gaming companies & startups. They are busy coming up with newer and popular games which appeals to their audiences. Issues of profits, growth, user-base, market capture, freshness & appeal, newer products, marketing, recruitment, etc. are among many of the organizational concerns on their minds.

Fluidity on the taxation front is giving gitters to the whole industry. A consistent and steady policy from the government is the need of the hour. At the same time social & heath issues like addiction, isolation, mental health issue are also serious concerns of the entire gaming industry. 

Several technological interventions like compulsory lay off time after a certain period of playing, etc are being brought in. As well as education & health interventions. While there seems to be a general acknowledgement of the issues, individual solutions may differ yielding varying results. A regulatory/ governance model is also needed here. Self-regulatory models while promising may/ may not work. Better is to institute a proper governance framework with checks & balances in place.

Finally, for what makes gaming worthwhile look at the success of Monopoly Go, a digitized version of the children's board game. The game is so popular that it  touched $1 Bn in revenues within 7 months of release & doubled that over the next 3 months. With that in mind, keep an eye out on all the action from the gaming world!



Tuesday, December 5, 2023

Pledge to Donate Organs

Across the nation there is a long list of patients waiting for matching organs such as liver, lungs, kidneys, etc to be transplanted from deceased donors. Yet the number of deceased donors & their families coming forward to donate organs is very small. 

There may be various reasons for the low numbers like awareness, knowledge, traditional, societal/ family norms, beliefs, etc. More importantly time is of essence when transplanting from deceased donors. All parties involved need to be on mission mode alertness and promptness. Any rethink or delay at such a critical stage can make the entire process futile. Always remember that transplanted organs can save lives!

The National Organ Transplant Organization (NOTTO) is the central body responsible for registering and managing organ donations. NOTTO has made it easy for individuals to pledge to donate their organs after demise online. The pledge as taken by the deceased while living makes it a lot smoother to conduct the transplant after their demise.

Others like MohanFoundation are also busy building awareness around organ donations. They host regular events and sessions on the topic across corporate houses, educational institutes, health facilites and so on and online.

To know more & to Pledge check out - Mohan Foundation Pledge, Hospitals accepting donations across cities, Army AORTA.

Saturday, November 11, 2023

Starting off with Databases

A note shared with a friend on getting started with DB on a Windows env. Putting it up for the larger audience.

1) Try MySql (or Postgre) DB Online via Browser:

• https://onecompiler.com/mysql/3zt5uh4dc
• https://www.w3schools.com/mysql/mysql_exercises.asp
• https://www.w3schools.com/mysql/exercise.asp
• https://www.mycompiler.io/online-mysql-editor
• https://www.mycompiler.io/new/mysql
• https://extendsclass.com/postgresql-online.html


2) Install MySql on Windows:

• https://dev.mysql.com/downloads/installer/


3) Run Windows Virtual Machine (VM) with a MS Sql DB installed in that VM:

3.1) Use either VMPlayer or VirtualBox as the virtualization software

VMWare Player: https://www.vmware.com/in/products/workstation-player.html
Oracle VirtualBox: https://www.virtualbox.org/wiki/Downloads


3.2) Download the corresponding VM:

• https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/


3.3) Login to the VM & install MS Sql server in it

Hope it helps.

Friday, August 18, 2023

On Patents

Quick notes on Patents. (To be improved over time )...

On Patent Search:

  • Google patent search doesn't cover India
  • For India: https://iprsearch.ipindia.gov.in/PublicSearch/
  • For World (includes India): https://worldwide.espacenet.com/patent/


For startup'ish low cost offerings:

  • https://www.onlinelegalindia.com/services/permanent-patent/
  • https://www.legalkart.com/legal-consultants-for-startups/intellectual-property/patent
  • https://intellectvidhya.com/patent-landing-page/
  • https://www.legalraasta.com/nation/patent_filing.html
  • https://ipflair.com/#Patentcalculator
  • https://www.zatapult.com/
(Note: Just listing them out as a ready reckoner. No experience with any of them. Please do your due diligence & check out everything thoroughly.)

 



Friday, September 30, 2022

Presentation to Image Converter

This article explains how to convert PPT/ Impress (Libreoffice) presentation file(s) to JPG Image(s) in a *NIX (Ubuntu) environment. Presentation file can be made up of several (N) slides which after final conversion yields one JPG image per slide (N JPG files).

Prerequisites software:

  • libreoffice
  • pdftoppm

 Conversion Process:

   PPT            --->     PDF           ----> JPG

   (N-slides)          (N-pages)          ----> JPG

                                                           .....

                                                              ----> JPG (N-JPG files, one per slide)

Steps to Convert

1. Convert PPT file to PDF

$ libreoffice --convert-to pdf "fileName.ppt" --outdir "outputFolderName"

(Note: For the given input file name fileName.ppt, this script creates an output file fileName.pdf in the specified outdir.)

2. Convert PDF to JPG

$ pdftoppm -jpeg -r 200 "outputFolderName/fileName.pdf" "imagePrefix_"

(Note: Image prefix is a text that can be added to the image file names.)

Finally, these steps are put to a shell script to allow it to convert all Presentations newly added to a folder inside called "PRESENTATIONS/Active/" in the user's Home folder.

Sunday, February 20, 2022

OCR

Tesseract that "quirky command-line tool that does an outstanding job" (credit A. Kay) truly works well.  Give it a shot whenever you get the opportunity.

Sample commands for ref:

  •    tesseract IMG-1.jpg IMG-1 --psm 4
  •    tesseract -l eng IMG-2.jpg b1


Monday, January 10, 2022

Water Crisis

At a time when major cities across India like Bangalore and Delhi are experiencing a major water crisis, critical interventions are the need of the hour. For what could work one should look at fellow nation South Africa for their handling the Day Zero crisis. The day when there is no more portable water available for use by the citizen. Here's some related coverage "Day Zero: Where Next?" (https://www.nationalgeographic.com/science/article/partner-content-south-africa-danger-of-running-out-of-water) & "Bengaluru is dying of thirst because it’s drinking its own Kool-Aid" (https://the-ken.com/the-nutgraf/bengaluru-is-dying-of-thirst-because-its-drinking-its-own-kool-aid/).

Reverse Osmosis (RO) water purifiers are both a boon & bane for the average household. Whith supply water TDS remaining way way above the palatable levels, ordinary non-RO basic filteration machines are rendered useless. But then RO machines end up throwing away waste water to the levels of about 5 - 10 litres (depending upon various factors) for every litre of drinking water purified. A criminal waste of the precious resource!


RO Water Recycling Bucket


Now we've been recycling the RO Waste water for other household cleaning, watering, etc purposes. This process might take you back in time to the days of filling up buckets of water from the supply line (a reality for many to this day), well, etc for use. Though a bit cumbersome this recycling bit works. In a span of one day we may be able to collect about 2 - 3 buckets (30 - 40 litres) of water which would otherwise have gone down the drain. All said and done, it's well worth the effort!


Monday, October 4, 2021

Unlimited Hassles

There's been a huge churn within the telecom sector in India over the last couple of years. These were precipitated by the laws and taxation policies in force, along with some rampant noncompetitive predatory practices, rock bottom pricing, m & a, essential upgrades to technology and infrastructure and so on, alongside a high growth of mobile phone & internet users (100+ crores) in the country.

The last factor helped bridge the connectivity and digital divide between the rural and the urban parts of the nation. Users of all age groups from all over came on board and got hooked on to social media sites and chatting apps. Online meetings, business communications, schooling, banking and payments, etc. went the mobile apps route.

While things were good for a while, a flip side to the story emerged soon. The large scale adoption was brought on by unsustainable predatory pricing by the players, esp. the new entrants. Which was closely followed by lowering of prices by the rest of the players. This race to the bottom, as expected, led to sinking of all but the most financially solvent ones. While some exited, others merged, the rest continue to struggle to stay afloat. An upward revision of prices therefore seems like the only way out of this mess.

On the other hand, a rise in prices will likely result in a drop in the number of users, particularly from the marginalized and weaker sections of the population. Perhaps a study is in order (or already done) which shows the impact per thousand (or lakh) users for every rupee (or ten) increase in prices. This mobile inclusivity for the citizens gained at long last must not be lost at any cost. Incorrect policies, corporate practices, profit motives, etc. of the past should not result in the nation regressing on the digital inclusivity front.

A sure shot Catch-22 for the policy makers from the sector: 

- To save the telecom players (via upward price revisions), or

- To preserve/ promote digital inclusivity for the citizens (particularly for the vulnerable).

One option that can be considered is to relook at the telecom pricing model. Telecom players these days offer various "Unlimited Plans". These have bundled unlimited data and call time (with daily sub-limits of a few gigs, minutes, etc.). These are therefore among the most popular plans and have led to an explosion of daily usage. People no longer care about the usage, while calling or using the internet/ data packs. As a result mobile bandwidths are practically choked all through the day. Poor quality services including frequent call drops, false rings, slow data connections are a menace for everyone. Plus there is also the adverse impact on the environment due to constant energy wastage happening at the level of the devices, network, switches, mobile towers, and so on.

The telecom pricing model in the past was the much more sensible "Pay-As-You-Go" model. Just like other shared basic need utilities available in a limited supply such as water, electricity, etc. telecom bandwidth (service) should also revert to the standard pay-as-you-go. This prevents wastage and allows a much fairer distribution of constrained resources.

There is a flat/ fixed nominal monthly subscription charge, and a variable usage cost billed per unit. Additionally by having separate consumption slabs, heavy/ corporate users can be made to pay more (as per a high cost slab), while the normal/ light users allowed to pay less. Thereby, making it easy on the pocket of the normal user and yet profitable for the telecom players.

The allied benefit of pay-as-you-go pricing will be that the number of mindless forwards, fakes, misinformation/ disinformation will go down, if not entirely disappear. Most people would be averse to spending even a few rupees daily towards the barrage of forwards and fakes. Resulting in a socially better and environmentally healthier world to live in!

Finally, to ensure inclusivity for the needy segments of the society and so that nobody gets left out, a separate "Janta (Citizen) Mobile Plan" could be introduced & a direct-to-account bill subsidy constituted. These little changes along with other significant ones on the corporate policy, laws, taxation and fair trade practices sides will ensure that India regains its lost ground in the telecom sector.

Thursday, June 3, 2021

Installing Canon LBP2900 Printer on Ubuntu 20.04

The instructions for installing Canon LBP2900 Printer on Ubuntu 20.04 remain pretty much the same as given in an earlier post about Installing Canon LBP2900 Printer on Ubuntu 16.04. The few differences seen this time around are the following:

1) Newer version of LINUX-CAPT-DRV-v271-UKEN: Download link as mentioned on the Canon Support page.

The tar ball includes 64-bit installers: cndrvcups-common_3.21-1_amd64.deb & cndrvcups-capt_2.71-1_amd64.deb (also includes rpm files & 32-bit versions).

2) Depdency on libglade2-0: 

$ sudo apt-get install libglade2-0

3) Workaround for CAPT 64-bit OS issues linking to 32-bit libraries: 

As mentioned in the earlier post, there are certain dependencies for CAPT running in 64-bit OS in linking to 32-bit libraries. Even though, the two deb files (mentioned above) get installed without the dependencies, peculiar behaviour/ error message is seen when the dependecies are missing. On the final step to View Status of printer on captstatusui, an error message is shown: "Check the DevicePath of /etc/ccpd.conf"

The solution to the problem is to simply check the missing dependencies:
(Check for error messages such as not executable, not found, etc.)

$ ldd /usr/bin/captfilter

$ ldd /usr/bin/capt* | sort | uniq | grep "not found"


In case of error install the missing dependencies:

$ sudo apt-get install libc6:i386 libpopt0:i386

$ sudo apt-get install zlib1g:i386 libxml2:i386 libstdc++6:i386


4) Follow the remaining instructions of the earlier post:

=> Install CAPT printer driver (downloaded .deb files mentioned above)

=> Add printer to the system: (Command line lpadmin or system-config-printer or UI System Settings > Printers):
 
=> Add printer to ccpadmin

=> View status of printer on captstatusui (should show "Ready to Print" if installed correctly)

=> Finally, print test page to finish!


 

Wednesday, March 31, 2021

Flip side to Technology - Extractivism, Exploitation, Inequality, Disparity, Ecological Damage

Anatomy of an AI system is a real eye-opener. This helps us to get a high level view of the enormous complexity and scale of the supply chains, manufacturers, assemblers, miners, transporters and other links that collaborate at a global scale to help commercialize something like an Amazon ECHO device.

The authors explain how extreme exploitation of human labour, environment and resources that happen at various levels largely remain unacknowledged and unaccounted for. Right from mining of rare elements, to smelting and refining, to shipping and transportation, to component manufacture and assembly, etc. these mostly happen under in-human conditions with complete disregard for health, well-being, safety of workers who are given miserable wages. These processes also cause irreversible damage to the ecology and environment at large.

Though Amazon Echo as an AI powered self-learning device connected to cloud-based web-services opens up several privacy, safety, intrusion and digital exploitation concerns for the end-user, yet focusing solely on Echo would amount to missing the forest for the trees! Most issues highlighted here would be equally true of technologies from many other traditional and non-AI, or not-yet-AI, powered sectors like automobiles, electronics, telecom, etc. Time to give a thought to these issues and bring a stop to the irreversible damage to humans lives, well-being, finances, equality, and to the environment and planetary resources!

Monday, March 29, 2021

Doing Better Data Science

In the article titled "Field Notes #1 - Easy Does It" author Will Kurt highlights a key aspect of doing good Data Science - Simplicity. This includes first and foremost getting a good understanding of the problem to be solved. Later among the hypothesis & possible solutions/ models to favour the simpler ones. Atleast giving the simpler ones a fair/ equal chance at proving their worth in tests employing standardized performance metrics.  

Another article of relevance for Data Scientists is from the allied domain of Stats titled "The 10 most common mistakes with statistics, and how to avoid them". The article based on the paper in eLife by the authors Makin and Orban de Xivry lists out the ten most common statistical mistakes in scientific research. The paper also includes tips for both the Reviewers to detect such mistakes and for Researchers (authors) to avoid them.

Many of the issues listed are linked to the p-value computations which is used to establish significance of statistical tests & draw conclusions from it. However, its incorrect usage, understanding, corrections, manipulation, etc. results in rendering the test ineffective and insignificant results getting reported. Issues of Sampling and adequate Control Groups along with faulty attempts by authors to establish Causation where none exists are also common in scientific literature.  

As per the authors, these issues typically happen due to ineffective experimental designs, inappropriate analyses and/or flawed reasoning. A strong publication bias & pressure on researchers to publish significant results as opposed to correct but failed experiments makes matters worse. Moreover senior researchers entrusted to mentor juniors are often unfamiliar with fundamentals and prone to making these errors themselves. Their aversion to taking criticism becomes a further roadblock to improvement.  

While correct mentoring of early stage researchers will certainly help, change can also come in by making science open access. Open science/ research must include details on all aspects of the study and all the materials involved such as data and analysis code. On the other hand, at the institutions and funders level incentivizing correctness over productivity can also prove beneficial.

Saturday, February 20, 2021

Debit Card to Bank Account Transfer

A feature missing from the Bank/ FinTech value chain is Domestic Debit Card (DC) to Domestic Bank A/c (BA) transfer. With wide proliferation of debit cards, payment gateways and POS vendors all provide C2B payments through these channels. Debit card to Bank A/c transfer doesn't exist, which would instead be a C2C/ P2P provision (via regulated payment intermediaries).

There ought to be very valid reasons for the same such as the regulator disallowing it, security & fraud considerations, transaction fees & gateway charges, error handling & reversal mechanism, and so on. 

Alternatives such as bank transfers provisions such as NEFT, RTGS, etc. and UPI mobile-apps based P2P transactions exist. Even for these modes all the issues mentioned above hold true yet the solutions were allowed to run and mature over time. So why not DC to BA?

If ever such a feature were to be rolled out by Banks/ FinTechs then all that customers would need is a single web page on the service provider/ Bank website to capture:

  • Debit Card details (Card no, Validity, Name, etc.) 
  • Recipient Details (A/c No, IFSC, Name)
  • Amount to Transfer

Next, the back-end gateway systems would:

  • Verify the details, authenticate the transactions, perform due handshakes for processing, etc.
  • If all ok, proceed to standard OTP based validation for the DC holder
  • Finally, debit the sender's DC (linked Bank A/c) & credit the receiver's Bank A/c

That's about it!


Monday, February 15, 2021

Parental Controls for Securing Online Usage by Children

As explained in the past, various safety features such as family shield filters from providers like OpenDNS , Cloudflare and others, DNS Over Https (DoH)HTTP Strict Transport Security (HSTS) can be used for a hassle free safe browsing across devices for members of the family. To additionally secure and regulate the usage for young kids Parental Control features and tools can be employed on devices and networks being accessed by children.

Parental Controls are available from day one across most device operating systems (OS) such as Android, iOS, and so on. All that the parent then needs to do, is to log in to the device using his/ her credentials and indicate to the device (OS) that the user of the device is a child and switch ON parental controls. Once that's done, the parental controls will get activated and only allow specific apps to run (apps white listed by the parent) while disallowing all others, and also filter out potentially harmful content from various sites and resources online.

Conceptually, that's pretty much all that there is to Parental Controls! For more info you can check out online resources such as these by Vodafone, VI and Google for a better understanding and setting-up parental controls to protect your kids online.

Monday, July 20, 2020

Inclusive Online Education

In today's times maintaining physical/ social distancing is the new way of living. While most of our activities will have to be done from home re-adjustments will be required to a few to shift them over to the digital channels.

Education is one key human endeavor which will progressively be done online. An assessment of the current state of online education here in India shows that while some resources do exist there's a lot that needs to be done to make online education viable and effective for students, particularly for those from the marginalized sections of the society.

Various government bodies connected with education in India such as CBSE, NCERT, AICTE, UGC, NIOS, CIET, CEC, MHRD, etc. have over the years made attempts towards providing distance education, e-learning & MOOCs, digitization of books and materials, content delivery via various digital plaforms & tv , youtube, video conferencing and so on. These freely available resources  can be good starting points for aggregating and rolling out well thought out standardized content/ tools for the students. Parallely, availability of content in regional languages and localization can be fast paced.

On the other side are the new world EduTech startup companies that are making lots of progress in the technology driven online education space. Companies such as Byjus, Vedantu, Khan Academy, TOI etc. are now household names thanks to their big advertising push. The target student pool though seems to be the well to do convent/ public school student with the means to pay for the services.

A good grasp over language and access to internet & good mobile and computing devices are pre-requisites for using these platforms well. While absence of such novelties in the lives of students from the poor and marginalized sections of society make the platforms out of reach of such students. Perhaps, it's time for socially conscious EduTech startups to come-forth to bridge the digital divide!
 
Update 1 (29-Mar-21): 
 
- Check out the discussion on EduTech at India Economic Conclave 21 between
Vineet Nayar(Founder & Chairman, Sampark Foundation) & Ashish Jhalani (CMO (Global), Square Panda)

Thursday, May 7, 2020

Ffmpeg - Swiss Army Knife of Video/ Audio Editting

Ffmpeg is a fantastic video and audio converter to edit & create video & audio files. As is typical of *nix command line tools, ffmpeg has several options that need to be correctly configured to be able to use the tool properly to edit videos or audios.

Architecturally, ffmpeg works with streams of video, audio, images or other data files that are passed through various reader/ writer (demuxer/ muxer) and encoder/ decoder layers for the editing and creating video and audio files:


Image Credit: Official Ffmpeg Linux Manual

The command prompt may be a little overwhelming to  start off, but a little playing with the tool shows reveals its immense potential. The official documentation page & Linux manual has a few examples to get you started.

Beyond this there are several online resources, blogs and articles such as this, this, this & this, etc. which have listed down the different ffmpeg commands with options. On the other hand, for those averse to the shell prompt, there are several GUI tools written on top of ffmpeg which can be explored.


Friday, April 17, 2020

Analysis of Deaths Registered In Delhi Between 2015 - 2018

The Directorate of Economics and Statistics & Office of Chief Registrar (Births & Deaths), Government of National Capital Territory (NCT) of Delhi annually publishes its report on registrations of births and deaths that have taken place within the NCT of Delhi. The report, an overview of the Civil Registration System (CRS) in the NCT of Delhi, is a source of very useful stats on birth, deaths, infant mortality and so on within the Delhi region.

The detailed reports can be downloaded in the form of pdf files from the website of the Department of Economics and Statistics, Delhi Government. Anonymized, cleaned data is made available in the form of tables in Section Titled "STATISTICAL TABLES" in the pdf files. The births and deaths data is aggregated by attributes like age, profession, gender, etc.

Approach

In this article, an analysis has been done of tables D-4 (DEATHS BY SEX AND MONTH OF OCCURRENCE (URBAN)), D-5 (DEATHS BY TYPE OF ATTENTION AT DEATH (URBAN)), & D-8 (DEATHS BY AGE, OCCUPATION AND SEX (URBAN)) from the above pdfs. Data from for the four years 2015-18 (presently downloadable from the department's website) has been used from these tables for evaluating mortality trends in Delhi for the three most populous Urban districts of North DMC, South DMC & East DMC for the period 2015-18. 

Analysis




1) Cyclic Trends: Data for absolute death counts for period Jan-2015 to Dec-2018 is plotted in table "T1: Trends 2015-18". Another view of the same data is as monthly percentage of annual shown in table "T-2: Month/ Year_Total %".




Both tables clearly show that there is a spike in the number of deaths in the colder months of Dec to Feb. About 30% of all deaths in Delhi happen within these three months. The percentages are fairly consistent for both genders and across all 3 districts of North, South & East DMCs.

As summer sets in from March the death percentages start dropping. Reaching the lowest points below 7% monthly for June & July as the monsoons set in. Towards the end of monsoons, a second spike is seen around Aug/ Sep followed by a dip in Oct/ Nov before the next winters when the cyclic trends repeat.


  


Trends reported above are also seen with moving averages, plotted in Table "T-3: 3-Monthly Moving Avg", across the three districts and genders. Similar trends, though not plotted here, are seen in the moving averages of other tenures (such as 2 & 4 months).

2) Gender Differences: In terms of differences between genders, far more deaths of males as compared to females were noted during the peak winters on Delhi between 2015-18. This is shown in table "T4: Difference Male & Female".




From a peak gap of about 1000 in the colder months it drops to about 550-600 range in the summer months, particularly for the North & South DMCs. A narrower gap is seen the East DMC, largely attributable to its smaller population size as compared to the other two districts.






Table "T5: Percentage Male/ Female*100" plots the percentage of male deaths to females over the months. The curves of the three districts though quite wavy primarily stay within the rough band of 1.5 to 1.7 times male deaths as compared to females. The spike of the winter months is clearly visible in table T5 as well.    

3) Cross District Differences in Attention Type: Table "T6: Percentage Attention Type" plots the different form of Attention Type (hospital, non-institutional, doctor/ nurse, family, etc.) received by the person at the time of death.




While in East DMC, over 60% people were in institutional care the same is almost 20% points lower for North & South DMCs. For the later two districts the percentage for No Medical Attention received has remained consistently high, the South DMC being particularly high over 40%.

4) Vulnerable Age: Finally, a plot of the vulnerable age groups is shown in table "T7: Age 55 & Above". A clear spike in death rates is seen in the 55-64 age group, perhaps attributable to the act of retirement from active profession & subsequent life style changes. The gender skewness within the 55-64 age group may again be due to the inherent skewness in the workforce, having far higher number of male workers, who would be subjected to the effects of retirement. This aspect could be probed further from other data sources.







Age groups in-between 65-69 show far lower mortality rates as they are perhaps better adjusted and healthier. Finally, a spike is seen in the number of deaths in the super senior citizens aged 70 & above, which must be largely attributable to their advancing age resulting in frail health.

Conclusion

The analysis in this article was done using data published by the Directorate of Economics and Statistics & Office of Chief Registrar (Births & Deaths), Government of National Capital Territory (NCT) of Delhi annually on registrations of births and deaths within the NCT of Delhi. Data of mortality from the three most populous districts of North DMC, South DMC and East DMC of Delhi were analysed. Some specific monthly, yearly and age group related trends are reported here.

The analysis can be easily performed over the other districts of Delhi, as well as for data from current years as and when those are made available by the department. The data may also be used for various modeling and simulation purposes and training machine learning algorithms. A more real-time sharing of raw (anonymized, aggregated) data by the department via api's or other data feeds may be looked at in the future. These may prove beneficial for the research and data science community who may put the data to good use for public health and welfare purposes.

Resouces: 

Downloadable Datasheets For Analysis:

Friday, February 28, 2020

Defence R&D Organisation Young Scientists Lab (DYSL)


Recently there was quite a lot of buzz in the media about the launch of DRDO Young Scientists Lab (DYSL). 5 such labs have been formed by DRDO each headed by a young director under the age of 35! Each lab has its own specialized focus area from among fields such as AI, Quantum Computing, Cognitive Technologies, Asymmetric Technologies and Smart Materials.

When trying to look for specifics on what these labs are doing, particularly the AI lab, there is very little to go by for now. While a lot of information about the vintage DRDO Centre of AI and Robotics (CAIR) lab is available on the DRDO website, there's practically nothing there regarding the newly formed DRDO Young Scientists Lab on AI (DYSL-AI). Neither are the details available anywhere else in the public domain, till end-Feb 2020 atleast. While these would certainly get updated soon for now there are just these interviews with the directors of the DYSL labs:

  • Doordarshan's Y-Factor Interview with the 5 DYSL Directors Mr. Parvathaneni Shiva Prasad, Mr. Manish Pratap Singh, Mr. Ramakrishnan Raghavan, Mr. Santu Sardar, Mr. Sunny Manchanda







  • Rajya Sabha TV Interview with DYSL-AI Director Mr. Sunny Manchanda





Wednesday, February 26, 2020

Sampling Plan for Binomial Population with Zero Defects

Rough notes on sample size requirement calculations for a given confidence interval for a Binomial Population - having a probability p of success & (1 – p) of failure. The first article of relevance is Binomial Confidence Interval which lists out the different approaches to be taken when dealing with:

  • Large n (> 15), large p (>0.1) => Normal Approximation
  • Large n (> 15), small p (<0.1) => Poisson Approximation
  • Small n (< 15), small p (<0.1) => Binomial Table

On the other side, there are derivatives of the Bayes Success Run theorem such as Acceptance Sampling, Zero Defect Sampling, etc. used to work out statistically valid sampling plans. These approaches are based on a successful run of n tests, in which either zero or a an upper bounded k-failures are seen.

These approaches are used in various industries like healthcare, automotive, military, etc. for performing inspections, checks and certifications of components, parts and devices. The sampling could be single sampling (one sample of size n with confidence c), or double sampling (a first smaller sample n1 with confidences c1 & a second larger sample n2 with confidence c2 to be used if test on sample n1 shows more than c1 failures), and other sequential sampling versions of it. A few rule of thumb approximations have also emerged in practice based on the success run techique:

  • Rule of 3s: That provides a bound for p=3/n, with a 95% confidence for a given success run of length n, with zero defects.

Footnote on Distributions:
  • Poisson confidence interval is derived from Gamma Distribution - which is defined using the two-parameters shape & scale. Exponential, Erlang & Chi-Squared are all special cases of Gamma Distrubtion. Gamma distribution is used in areas such as prediction of wait time, insurance claims, wireless communication signal power fading, age distribution of cancer events, inter-spike intervals, genomics. Gamma is also the conjugate prior of Bayesian statistics & exponential distribution.

Wednesday, October 23, 2019

Over-Smart World

BlackMirror BlackMirror On the Wall
Who's the Smartest of them All?
 
Smart smart Alexa, 
Smart smart TV, 
Smart smart Watch,
Smart smart This,
And smart smart That

They never sleep?
Keep always an eye (& ear) on me.

BlackMirror BlackMirror On the Wall
Who's the Smartest of them All?

Thursday, October 3, 2019

Firefox Normandy

Firefox through the Normandy feature provides an option for unsolicited/ automagic updates to default values of a Firefox (targetted) instance. For more on the risk this poses take a look at the ycombinator threads.

To turn off Normandy in Firefox use the advanced settings route: about:config > app.normandy.enabled = false.

Update 1 (23-Oct-19):
 - Principally Mozilla (Firefox) have always been in favour of user privacy.

Saturday, September 21, 2019

Last Petrol Car

In the year 2024 my present BS-III Hyundai petrol (BS-III) hatchback would reach its end of life, 15 years after its first drive out of the showroom. Given all the buzz from the Electric Vehicle (EV) space, this would very likely be my last petrol car. At some level, most of us have next to zero attachment with the fuel that powers the vehicle under the hood (petrol, cng, electricity, etc.). What we care about is that the new vehicle shouldn't be a downgrade in terms of reliability, comfort, features, looks, pricing, drivability, power, pickup, etc and an increase in terms of purchase & running costs. 

Battery operated EVs seem to be getting better by the day. There's good traction seen in the three-wheelers (battery operated autos/ totos) space. Two- & four-wheelers are likely to hit mass markets soon, with pricing that would be lucrative (perhaps tax incentivized). Further, widespread infrastructural & service support need to be introduced to give people the confidence to switch to EVs.

Yet, at the moment, EV technologies - battery, chargers, fire & safety protocols, instrumentation, cabling & connectors, etc. - are at early-to-mid maturity level. Driving range per charge is about 100 Kms for the entry segment cars which is not enough. It's quite common for people to drive ~150 Kms daily for work. On highways, the range could be much more. So a sub-300 Km range would simply not do!

At the same time, the mass market pricing levels (INR 3 to 6 lacs) should not be breached in any way. The existing coverage of mechanics & service centres of various manufacturers (Maruti, Hyundai, Mahindra, Tata, etc.) needs to be upgraded to support EVs as well.

Reliable electricity remains a constraint in most cities including the metros. On the generation side, renewables would need a wider push. Residential solar rooftop set-ups could be one area of focus. Through such set-ups, individual households & complexes could achieve self-sufficiency for their growing energy needs, including the EV burden/ load (@20-30 Units for full charge per vehicle X 30 days = 600-900 units per vehicle per month). Standard practices to popularize rooftop solar set-ups employed the world over such as PayGo models, incentives/ tax breaks, quality controls, support & maintenance, etc. should be introduced here as well. If possible, it would be great to have the EVs themselves equipped with solar panels on the body to auto-charge whenever required under direct sunlight. Eagerly waiting for these clean green technologies to evolve and make inroads very soon!

Update 1 (09-Oct-19):
 - An assessment of the current state of EV adoption in India by Business Standard.

Update 2 (23-Oct-19):
 - Bajaj Chetak to be relaunched in an Electric avatar.
 - Blu-Smart all electric cabs visible on Delhi roads.

Thursday, September 19, 2019

Renewable Energy In India

India holds great potential in the renewable energies space. We have ample opportunities to generate all our present and future energy needs from sources like solar, wind, water and biomass.

From an energy generation capacity from renewables pegged at ~60 GW (in 2017) we are targetting to reach about 175 GW (100 GW Solar, 60 GW wind, 10 GW biomass, 5 GW small hydro power) by 2022. Which would be close to 50% of our entire energy needs. With ground work for mass adoption of Electric Vehicles (EV) getting traction, our demands for power and generation from renewables will need to scale up even further. To the extent that we may become energy surplus one day and be able to export to the neigbourhood. For a sneak peak into the state of the art from the world of renewables, head over to the Renewable Energy India (REI) Expo 2019 currently underway at the Knowledge Park II, Greater Noida.

The REI-2019 has exhibitors from leaders in the renewables space such as China, Bangladesh, France, Germany, India, Israel, Netherlands, Saudi Arabia, Singapore, Slovakia, South Korea, Taiwan, Tunisia, UK, USA, Vietnam, etc. They are showcasing their product portfolios from solar & wind power devices to installations on floating & permanent structures, from switching & grid apparatus to connectors, from inverters & batteries to EVs, and more. Expo timings are from 10 am to 6 pm. Walk-in as well as online registrations are allowed. Go see the future!

Update 1 (21-Sep-19):
- Listen to what Greta Thrunberg has to say & check out her zero-carbon boat

Update 2 (23-Oct-19):
- Coal to continue powering India's energy requirements for decades - Swaminomics