Anantha Kumaran random thoughts

Visualization of backoff functions

Failures are inevitable in any system. How it should be handled varies from one system to another. In job processing systems, a common approach is to retry the failed jobs for a fixed number of times before they are considered as permanent failures. A backoff function is used to determine the wait time between successive retries.

Let’s look at a simple backoff function, which retries after a fixed wait time – 5 minutes in this case.

function constant(retryCount) {
   return 5 * 60;
}

Let’s assume there are 100 job failures, the chart below shows when each of the jobs would be retried again. Each dot represents a time at which a job is retried. The color is varied based on the retry count.

An exponential backoff function increases the wait time exponentially for each retry.

function exponential(retryCount) {
    var min = 3 * 60;
    var base = 2;
    return min + (Math.pow(base, retryCount) * 60);
}

The above two are pure functions, given an input they will always return the same output. If n jobs failed at the same time, then all the n jobs will be retried at the same time, which could cause thundering herd problem. A random component called jitter is normally added to fix this problem. The last component in the function below is a random jitter that is scaled based on the retry count.

function sidekiq(retryCount) {
    return Math.pow(retryCount, 4) + 15 +
        (Math.random() * 30 * (retryCount + 1));
}

The above function is good enough for most of the use cases. There are still some gaps where your job processing system would be idle. The below function tries to distribute the load evenly by increasing the randomness.

function between(a, b) {
    return a + (Math.random() * (b - a));
}
function buckets(retryCount) {
    var exp = 3;
    return between(Math.pow(retryCount, exp),
                   Math.pow(retryCount + 2, exp));
}

Although the load distribution is better than the previous version, the wait time between two retries starts to deviate a lot. I wonder if there are any stateless functions which could provide better distribution without much deviation in wait time.

Debugging cryptic errors

At a project I am working on, our backend system makes http requests to hundreds of different servers. It is sort of like webhook, http requests are made to the customer server with the payload whenever some specific events occur in our system. We were in the midst of migrating our systems from ruby to elixir.

During the test run, we started to get a cryptic error for some small number of domains.

HTTPoison.get("https://piay.iflix.com", [], [proxy: {'10.128.10.16', 3128}])
# => timestamp=2017-01-02T04:11:00.816Z level=error message= SSL: :hello:ssl_alert.erl:88:Fatal error: handshake failure

# => {:error, %HTTPoison.Error{id: nil, reason: {:tls_alert, 'handshake failure'}}}

The error points out that the handshake phase during the ssl connection is failing. But it doesn’t say why it is failing.

To get more information, I tried to use ssl module directly.

:ssl.connect('piay.iflix.com', 443, [])
# => {:ok,{:sslsocket, {:gen_tcp, #Port<0.10658>, :tls_connection, :undefined}, #PID<0.325.0>}}

To my amazement, it worked. I was able to connect to this domain, but https request failed. Due to compliance reasons, we have to use a proxy server for all our outgoing http/https requests. We use HTTPoison library, which is a wrapper for hackney library. We already had some issues due to proxy. It seems like most of the users of the hackney library don’t use the proxy option, so some of the code paths related to proxy are not well tested. To make sure proxy is the problem, I made the request without proxy option and it worked.

Hackney uses connect tunneling method, which is quite simple. The http client sends CONNECT piay.iflix.com:443 to the proxy server, which in turn opens a tcp connection to piay.iflix.com:443. The proxy server will then relay whatever data sent by http client to the destination server. In case of https request, once the connection is established, hackney initiates the ssl protocol using ssl:connect method.

This looks quite simple, but still something is going wrong. The same ssl:connect succeeds when it is established directly, but not through the proxy server.

The dbg app provides text based tracing functionality. It can be used to trace function at various granularities, from all the functions in a module to a function with specific arguments. I started to trace all the function calls in ssl module, but it resulted in too much data for me to analyze properly. Then I started to read the source code of ssl app and started to trace a single function at a time and compared the arguments and the results of successful and failed connection.

(<0.572.0>) returned from ssl_handshake:client_hello_extensions/6 -> 
{hello_extensions,
 {renegotiation_info,
  {elliptic_curves,
   {1,3,
    132,
    0,
    30}]},
 undefined}


(<0.572.0>) returned from ssl_handshake:client_hello_extensions/6 -> 
{hello_extensions,
 {renegotiation_info,
  {elliptic_curves,
   {1,3,
    132,
    0,
    30}]},
 {sni,
  "piay.iflix.com"}}

After multiple trials and errors, I finally came across something interesting. The sni field was present in successful connection, but not in failed connection. The rest of the deduction was easy. The sni extension allows a single server to serve different certificates for different domains. During the initial handshake, the client has to provide the domain name along with other details.

If a host name is passed as the first param of ssl:connect, the sni extension is automatically enabled. For proxy request, the connection is established to the proxy server, which relays the data to the destination server. As the ip address of the proxy server is passed as the first param, sni extension was not enabled.

HTTPoison.get("https://piay.iflix.com", [], [proxy: {'10.128.10.16', 3128}, ssl: [server_name_indication: 'piay.iflix.com']])

The fix was easy. The sni extension has to be enabled explicitly. As always, more layers introduce more points of failures.

80TTA

Section 80TTA of Income Tax Act allows deduction up to ₹10,000 on the income earned as interest from a savings account. At interest rate for savings account, a principal of ₹ would generate ₹10,000 as interest in a year. The effective amount of interest you would make for the same principal in fixed deposit scheme would differ based on the tax slab (slab 1 - 10%, slab 2 - 20%, slab 3 - 30%) you belong to. You should use fixed deposit scheme as long as the interest rate is higher than , , for slab 1, slab 2 and slab 3 respectively.

Probability behind marriage horoscope

Most of the Indians do arranged marriage. One part of the process is the horoscope matching. The horoscope matching process itself varies based on language, religion, caste, location etc. I am going to dissect one such process which is mainly followed by people in south India. I don’t have much knowledge about horoscope or anything related to astrology, neither do I believe in it. So it is possible that I might have misunderstood some aspects of the process. So take anything I say with a grain of salt. I am using tamilhoroscope.in as the main source for the rules.

Porutham is a sub part of the matching process. Each individual porutham projects a specific aspect of a marriage life. For example Dhinam porutham projects the opinion and understanding between the couples in everyday life. There are 8 main poruthams namely Dhinam, Ganam, Mahendra, Rasi, Nadi, Yoni, Rajji and Vedhai. Every porutham need not match to be considered a successful match. A successful match can be succinctly described using boolean operators as

(Dhinam || Ganam) && (Mahendra || Rasi || Nadi) && Yoni && Rajji && Vedhai

There are 27 nakshatras and 12 rasis. A person’s rasi and nakshatra are calculated using their birth date and time. Each porutham can be described as a set of rules.

Dhinam

For each and every day to be a very special day for the couples they must have their match of Dhina porutham. Only this match will eliminate the difference of opinion and misunderstanding between the couples. This match is made as follows:

  1. If a remainder of 2, 4, 6, 8, 9 is obtained when you divide the counting of stars from female’s star to male’s star by 9 then this match is fulfilled.

  2. If the counting from female’s star to male’s star is 2, 4, 6, 8, 9, 11, 13, 15, 17, 18, 20, 22, 26, 27 then this match is fulfilled.

  3. If both male and female are of the same rasi or belong to different Phase of the same star then this match is fulfilled.

  4. If both male and female are of Bharani, Avittam, Sadhayam or Poorattathi stars then this match is not fulfilled. Both having any of the above stars for them should in fact avoid getting married.

Ganam

There are 3 Ganams as per astrology. They are Deiva Ganam, Manitha Ganam, Ratchasa Ganam. All the stars would definitely fall in any one of the above ganams. This match is viewed especially to arrive at the tolerance level between the couples. If the couples have this match in their horoscope then they will tolerate each other and avoid misunderstandings.

  1. Ashwini, Murugashreedam, Punarpoosam, Poosam, Hastham, Swathi, Anusham, Thiruvonam and Revathi are the stars belonging to Deiva ganam.

  2. Bharani, Rohini, Thiruvadirai, Pooram, Uthiram, Pooradam, Uthiradam, Poorattathi and Uthirattathi are the stars belonging to Manitha ganam.

  3. Karthigai, Aayilyam, Magam, Chittirai, Visakam, Kettai, Moolam, Avittam and Sathayam are the starts belongings to Ratchasa ganam.

A male and a female belonging to the same ganam or even if either belong to deiva ganam and other belong to manitha ganam they can get married.

If the male belong to deiva ganam and the female belong to ratchasa ganam then they do not have gana porutham on the contrary if the female belong to deiva ganam and the male belong to ratchasa ganam then this match is fulfilled.

Mahendra

Mahendra porutham is matched for puthira bakkiyam(blessed with a child). This match will hint the flow of the generations of that particular family.

If the counting from the female star to the male star is 4, 7, 10, 13, 16, 19, 22, 25 then this match is fulfilled.

Rasi

Rasi porutham is one of the many matches looked for vamsa viruthi and as per few sasthras this match will give all the happiness and the blessings to have a male child for the couples getting married.

If the counting from the female star to the male star

Is above 6 then matching is fulfilled

Is 8 then there is no match

Is 7 then the matching is excellent

Is 2, 6, 8, 12 then there is no match

Is 1, 3, 5, 9, 10, 11 then the matching is ok

Also Kumbam and Simam, Magaram and Kadagam are the rasis that do not match with each other.

Nadi

This match is being looked for to denote the matching between the blood groups of the male and female. Budhira Bakkiyam for the couples having this match is confirmed.

In general Nadi is divided into three divisions. If both male and female belong to the same Nadi then this match is not full filled. And if both Male and female are of different Nadi then this match is fullfilled.

Paarsuva Nadi - Ashwini, Thiruvathirai, Punarpoosam, Uthiram, Astham, Kettai, Moolam, Sathayam, Puratahi

Madhya Nadi - Barani, Mirugasrisham, Poosam, Puram, Sitthirai, Anusham, Puradam, Avitam, Uthiratathi

Samana Nadi - Karthigai, Rohini, Ayilam, Magam, Swathi, Visakam, Uthiradam, Thiruvonam, Revathi

Yoni

As per this match each and every star is identified as an animal. This match is not fulfilled if there is a enemity between the yoni(animals). And for fulfillment of this match there should not be enemity between the animals(yoni) matched.

Aswini - 🐴 Male Horse
Baraṇi - 🐘 Male Elephant
Karthikai - 🐐 Female Goat
Rohiṇi - 🐍 Male Snake
Mirugasīridam - 🐍 Female Snake
Thiruvādhirai - 🐶 Female Dog
Punarpoosam - 🐈 Female Cat
Poosam - 🐐 Male Goat
Ayilyam - 🐈 Male Cat
Magam - 🐀 Male Rat
Pooram - 🐀 Female Rat
Uthiram - 🐮 Male Cow
Astham - 🐃 Female Buffalo
Chithirai - 🐯 Female Tiger
Swathi - 🐃 Male Buffalo
Visakam - 🐯 Male Tiger
Anusham - Female Deer
Kettai - Male Deer
Mulam - 🐶 Male Dog
Puradam - 🐵 Male Monkey
Uthirādam - Male Mongoose
Tiruvōnam - 🐵 Female Monkey
Aviṭṭam - 🦁 Female Lion
Sadayam - 🐴 Female Horse
Puraṭṭādhi - 🦁 Male Lion
Uttṛṭṭādhi - 🐮 Female Cow
Revathi - 🐘 Female Elephant

Animals with Enemity

🐍 Snake x Mongoose
🐘 Elephant x 🦁 Lion
🐵 Monkey x 🐐 Goat
Deer x 🐶 Dog
🐀 Rat x 🐈 Cat
🐴 Horse x 🐃 Buffalo
🐮 Cow x 🐯 Tiger

Rajii

This is the most important match that is looked for while matching the horoscopes. For the persons getting married, even if nine out of ten matches are full filled and if this Rajji porutham is not fulfilled then it is not recommended for them to get married. Our ancestors have given such an importance for this match. Rajju is being divided into five parts.

Siro Rajii - Mirugasrisham, Sitthirai, Avittam

Kanda Rajii
Arohanam - Rohini, Astham, Thiruvonam
Avaraohanam - Thiruvathirai, Swathi, Sathayam

Uthara Rajii
Arohanam - Karthigai, Utharam, Uthradasm
Avaraohanam - Punarpoosam, Visaham, Purattathi

Uuru Rajii
Arohanam - Barani, Pooram, Pooradam
Avaraohanam - Poosam, Anusham, Utthiratathi

Paadha Rajii
Arohanam - Aswini, Magam, Mulam
Avaraohanam - Ayilam, Kettai, Revathi

Matches are looked for in such a way that if the stars of both male and female are not of the same Rajji then the female will lead the life as Dhirka Sumangali.

There are two divisions in the same Rajju like Arohanam and Avaraohanam. There is an opinion that even If the stars of the male and the female are of the same Rajju but belong to Arohanam and Avarohanam then the horoscopes are matched and proceeded for marriage.

Vethai

This Vethai Porutham denotes the power of Mangalya Kayiru and is one of the basic match that is being looked for. If both male and the female stars are Vethai then this match is not fullfilled and the others have Vethai match.

Ashwini - Kettai
Bharani - Anusham
Karthikai - Visakam
Rohini - Swathi
Thiruvathirai - Thiruvonam
Punarpoosam - Uthiradam
Poosam - Puradam
Ayilam - Moolam
Magam - Revathi
Puram - Uthiratathi
Uthiram - Uthiratathi
Hastham - Sathayam

Above are the stars that do not match with each other. And matching the above stars is not a good match.

Dhinam || Ganam

Mahendra || Rasi || Nadi

(Dhinam || Ganam) && (Mahendra || Rasi || Nadi) && Yoni && Rajii && Vethai

Visualization of Bit Reversal Ring

Bit reversal ring of size z could be defined as the reverse of the binary representation of each number x in the sequence 0 to z - 1.

When the size of the ring is 2n, the reversal just rearranges the numbers in the sequence. It doesn’t remove or introduce any new number to the list.

In the first two visualizations, line represents a interchange and the color of the line represents distance between the two numbers.

Two list of numbers a and b can be said as order equivalent if and only if a[i] < a[j] = b[i] < b[j] for every i and j.

An example would be [1, 5, 3] and [2, 6, 4]. The bit reversal operation preserves some of the order equivalence of sub groups within the ring. Each arch represents a group of numbers after bit reversal operation. Groups with same order share same color. The size of the group increases as you go from top to bottom. The size of the ring (4, 8, 16) increases as you go from left to right.

For example, the ring in the column 2 and row 3 represents the groups (0,4,2), (4,2,6), (2,6,1), (6,1,5), (1,5,3), (5,3,7), (3,7,0), (7,0,4).