Cannon Beach [Haystack Rock]

 Canada across the strait!![Port Angeles]

                                   Another one of those roads to perdition [Olympic National Park]

Turn up that heat [Heart O' the Hills road, Olympic National Park] 

Tunnel in a tunnel [Heart O' the Hills road, Olympic National Park]  

  Ridgy-Ridge [Hurricane ridge, Olympic National Park]

[Heart O' the Hills road, Olympic National Park]   

 [Elwha Rainforest, Olympic National Park]  

 [Lake Crescent, Olympic National Park]

Lost trees up-top [Rialto beach]

So I came across an interesting problem and the problem was essentially a modification of permutation problems that I had seen before. I did end up writing the brute force solution which was not good enough and had to revisit the problem for optimizing it.
Here is my best attempt at describing the problem -
Each token in a line is sent through a encryption module and gives out a transformed version of the token. The encryption module has some bugs in it and hence not all words may always get encrypted/transformed, find all possible representations of the same line so that the representations cover all possible permutations if each token was successfully transformed or not.

I hope I did not make the problem statement too confusing but to make things a bit more clearer, here is an example :-
Line -> “W1 W2"
“W1" transforms to a transformed word called “TW1”
“W2” transforms to a transformed word called “TW2”

Hence, considering that both words may not always get transformed, here are all the possible permutations of the above line.
Permutation1 -> “W1 W2”
Permutation2 -> “TW1 W2”
Permutation3 -> "W1 TW2”
Permutation4 -> “TW1 TW2”

Similarly, if the line was “W1 W2 W3” with the respective transforms being “TW1”, “TW2” and “TW3” then the possible permutations are :-
“W1 W2 W3”
“TW1 W2 W3”
“TW1 W2 TW3”
“TW1 TW2 W3"
“W1 W2 TW3"
“W1 TW2 W3"
“W1 TW2 TW3"
“TW1 TW2 TW3”

Solution 1(Brute force) :- This was the first approach I took to get things done.
Create a hashset or a treeset of results and go over each possible combination with the given token at hand. To top it off, I made the solution worse by making it recursive. So for example, if you take the first token in a three token line, then get all the possible combinations of the remaining two tokens and append them to W1 and TW1. Then, move onto W2, and get all possible permutations of (W1, TW1) and (W3,TW3) and attach them around W2 and TW2.
As can be seen, this solution is tedious and wasteful. We end up recalculating many of the already finished permutations and there needs to be an easy way to avoid that.
The worst case is when you have many tokens around the token at hand and one has to recursively calculate the combinations on both sides of the token and then again permutate with the given token. So, one creates three subproblems :-
i) Calculate permutation before the token(P1)
ii) Calculate permutation after the token(P2)
iii) Add in Permutations of P1,token,P2

I wish not to rewrite code for such a lame approach though the time complexity will be O(N X Maximum Number of Permutations before a token X Maximum Number of Permutations after a token) where N is number of tokens
which basically amounts to O(N!(N-1)!….1!).
Enough said about how bad it was.
Unfortunately, this method only worked until like 10 or 11 tokens on my laptop and my laptop gave up after that.

Solution 2(Use a tree/graph approach):-
This solution was thought about it so as to avoid recalculating a path already taken. Once all possible paths of a tree are taken, one can be assured that all possible permutations have been calculated. The way of thinking about this solution is to go have a directed graph with two start points W1 and TW1. And then do a Depth First Search for last token and its transform(in this case W3 and TW3). I think of this as a problem having two sources(W1and TW1) and two sinks(W3 and TW3).
Here is a diagram to explain the same.

fig - Graph Approach

The interesting part is that using this approach, I ended up using a little more space than I normally would to keep tabs of the nodes I have already visited.
This approach had O(V * E) time complexity which is O(N x 2N) which is essentially O(N^2) with O(2N) space complexity to maintain the visited nodes.

Solution 3(Using a binary bit based approach) :-
Thanks to Randy Gobbel for noting a pattern from the above graph and suggesting a uniquely simple approach without any extra space needed. Randy noted that each place in the resultant permutation has two possible values(W or TW). This is more or less like 0 or 1. Hence, the permutations amount for a three token line amount to -
000, 001, 010, 011, 100, 101, 110, 111
where 0 -> the token itself
          1 -> the token’s transform
As a result, this solution boils down to simply iterating over all possible values from 0 to 2 raised to the power of the number of tokens. Below is the code for the approach. The method signature involves a list of tokens and a Map/Dict containing transforms for each token.

1:  public List getPossiblePermutations3(List words, Map transformMap) { 
2:        if (words == null || words.isEmpty()) 
3:            return null; 
4:        List permutationList = new ArrayList<>(); 
5:        for (int i = 0; i < (int)Math.pow(2,words.size()); ++i) { 
6:            StringBuilder permutation = new StringBuilder(); 
7:            for (int j = 0; j < words.size(); ++j) { 
8:                //Check if bit is set 
9:                if((i & (int)Math.pow(2, j)) == 0) 
10:                    permutation.append(words.get(j) + " "); 
11:               else 
12:                    permutation.append(transformMap.get(words.get(j)) + " "); 
13:            } 
14:            //Trim out end spaces/blank characters 
15:            permutationList.add(permutation.toString().trim()); 
16:        } 
17:        return permutationList; 
18:    } 

The time complexity for this approach is O(N x 2^N) with no space implications.

Thanks for reading…

Problem :­

We have some types of documents which have to be deleted daily based on the date in them and if it has expired. An alert from another tool tells us that if such an old dated document appears in the search results which is filtered out by the end service.

So, for the past month, we saw that the alert was getting triggered every night between 12am and 7:15am and then it disappears (no more data from yesterday)

Remember, that these documents are deleted daily using the XDELETE query to the elasticsearch cluster.

Experiments :­

We run an hourly job to delete old data and logically, one expects the old data to disappear as soon as the first delete job is run after 12am, but we keep seeing the alert with the log showing that a result from yesterday was getting triggered.

I increased the job frequency to run every 15 minutes with a "refresh" on the index after each job and yet, the splunk alert would show up until 7:15am every day. The pattern that is seen is between 6am and 7am, the number of old documents begin to decrease and finally there are none after 7:15am. Even "optimize_expunge_deletes" does not decrease the time these documents show up in the elasticsearch results.

Possible causes :­

• The second last comment on this forum reveals that elasticsearch marks documents for deletes and by default, merges only happen on an index when there is 10% or more documents to be added or deleted. As we index daily, probably 7:15am is the time when a merge happens and the documents get deleted.

• Though, we see the docID in the logs, when searching for the documents in the elasticsearch cluster with the ID leads to no results and hence, this points to that different results shown from the head than from the client API.

Possible solution :­

Note, the index does not use TTL(time­to­live) value for the documents but as per the elasticsearch forums, people still see this issue after using TTL.

So supposedly, this is the way lucene and elasticsearch want to be efficient by limiting merges as a merge is an expensive operation from memory’s perspective.

The likely solution given by elasticsearch forum experts is to create new index and swap indices whenever possible(this means swapping indices everyday for us) as a new index has faster searches avoiding the deleted documents in the lookup (overkill !!).

Considering, most of the elasticsearch users talk about documents being present in the millions, sometimes it is not feasible to index and come up with a new index daily. So, an alternate short approach can be to update the document with a field of deleted set as true and filter out such documents.

Problem :­
A particular search query would show results sometimes and sometimes won't.

Root cause analysis :­

A pattern observed with the hits and misses was of 011100 occuring in a repeated way.

Elasticsearch by default delegates the queries amongst the nodes till it finds the results and hence, if you want to direct your query to a particular node, a "preference" parameter can be set to "nodeID"

The way to get a nodeID (the one that I found) is ­
curl ­XGET 'localhost:9200/_nodes'

wherein every node has a ID associated in the json header.
The nodeID can be used as ­
searchRequestBuilder.setPreference("_only_node:xyz");

Using this preference, one can narrow down the shard that was showing this inconsistent behaviour for the given document. The shard stats showed that it had the same number of "numDocs" and hence it seemed for some reason, the document was not searchable on that shard.

Found a similar issue in elasticsearch groups ­

Resolution :­

The elasticsearch logs did not show any issue on the nodes and that "refresh" and "flush" on the index did not produce any exception/error in the logs and did not resolve the issue.

As per the issue thread, people had recreated their indexes or stuck to using the primary shard(which wasnt an option as recreating the index would take a lot of time and hitting the primary shard only always gave us no hits)

The no­-downtime way was to bring the replica shards down to 0 and create them again. (which luckily fixed the issue for us)

curl ­XPUT 'localhost:9200/my_index/_settings' ­d '{
  "index" : {
    "number_of_replicas" : 0
  }
}'

Run an indexing job to fill back in the documents lost in the process.

Possible Causes :­

We had some downtimes in the cluster due to high CPU usage and maybe such outages would have caused an inconsistency in the shards. Though, we still kept getting consistent results for most of the queries we had.

So the guess is that any such issue can cause elasticsearch shards to become inconsistent and the resolution is variable/dependent on the number of documents in the index and the possible downtime that can be taken.

So every now and then one comes across some puzzle which makes you delve deep into number theory and though things appear weird at the start, they end up being deterministic... or precisely certainly solvable though in infinity !!

Here is the "Sinking the Submarine" problem -

[I got this puzzle from Alex Mikhailov and a little help from Yuval]

An enemy submarine is somewhere on the number line (consider only integers for this problem). It is moving at some rate (again, integral units per minute). You know neither its position nor its velocity. You can launch a torpedo each minute at any integer on the number line. If the the submarine is there, you hit it and it sinks. You have all the time and torpedoes you want. You must sink this enemy sub - devise a strategy that is guaranteed to eventually hit the enemy sub.

Solution :-

Looking at the problem, one feels a simple strategy can help us reach the solution. But then one goes into asking random questions, like 

- Do we know the initial position?

- Should we even care about the initial position?

Remember that the submarine does not have an acceleration. So atleast, some randomness has already been constrained. So, lets go about proving that the submarine can be sinked certainly.

In this puzzle, we have combination of values for velocity and displacement of submarine on the number line based on a function between initial position and velocity. Remember, that the speed(velocity) remains constant. So, the submarine would be travelling the same distance between time t0 to t1 as it would between time t1 and t2.

The possible values of velocity are 0,1,2,..... infinity

The possible values of starting displacement are 0,1,2,...infinity.

So, we need to traverse the combinations of starting displacement and velocity in such a way that we hit the submarine somewhere before infinity. Using a simple function, such as 
currentPostion = startingDisplacement + (time * velocity) 
we can brute force each combination value and sink the submarine before infinity.
Here is one of the lame ways to iterate over all possible combinations of velocity and time

Had some free time, so decided to code it in.

1: import java.util.Scanner;
2: 
3: public class SinkTheSubmarine {
4:
5: public static void printNumShotsRequired(long startPosition, long velocity) {
6:  if (startPosition == 0 && velocity == 0) {
7:   System.out.println("Submarine hit on 1 attempt"); 
8:   return;
9:  } 
10:  long subPosition = startPosition;
11:  long time = 1;
12:  long startDisp = 0;
13:  long vel = 0;
14:  long shotPosition = 0;
15:  boolean hit = false;
16:  boolean changeStartDisp = false;
17:  boolean changeVel = false;
18:  
19:  startDisp++;
20:  changeStartDisp=true;
21:  do { 
22:   System.out.println("startDisp:"+startDisp+" vel:"+vel);
23:   subPosition = startPosition + (time * velocity);
24:   shotPosition = startDisp + (time * vel);
25:   if (shotPosition == subPosition) {
26:    System.out.println("Submarine hit on " + (time + 1) + " attempt"); 
27:    hit = true;
28:   }
29:   if (changeStartDisp && !changeVel) {
30:    if (startDisp == 0) {
31:     //Move down
32:     vel++;
33:     changeStartDisp = false;
34:     changeVel = true;
35:    } else {
36:     //Move left bottom
37:     vel++;
38:     startDisp--;
39:    }
40:   }else if(!changeStartDisp && changeVel) {
41:    if (vel == 0) {
42:     //Move right
43:     startDisp++;
44:     changeStartDisp = true;
45:     changeVel = false;
46:    } else {
47:     //Move right up
48:     startDisp++;
49:     vel--;
50:    }
51:   }  
52:   time++;
53:  } while(!hit);
54:  return;
55: }
56: 
57: public static void main(String[] args) {
58:  Scanner scanner = new Scanner(System.in);
59:  System.out.println("Enter starting location of submarine:");
60:  long startstartDisplacement = scanner.nextInt();
61:  System.out.println("Enter velocity of submarine:");
62:  long velocity = scanner.nextInt();
63:  System.out.println(startstartDisplacement + " " + velocity);
64:  printNumShotsRequired(startstartDisplacement, velocity);
65: }
66:}

Remember, that if we had acceleration in the picture, then we would have to change our function with something like Newton's second law of motion -
s = ut + (at²/2)

and will have to iterate over combinations for starting displacement, velocity and acceleration and hence, even in that case, sinking the submarine is a certainty.

Burning up the rocks to find a way through [Location :- McArthur-Burney Falls]

Some science experiment. Sulphur mixing with boiled up water [Location - Lassen National Park]

What happens outside the visitor center, stays outside the visitor center [Location - Crater Lake National Park, Oregon]

Steps to the temple with Avalanche Gulch shining bright [Location - Bunny Flat trail, Mount Shasta]

For sometimes, there is no reference line to match when the expanse is so huge [Location - Mount Shasta]

 Because sometimes one is lazy enough to not go down [Location - Middle McCloud Falls]

The crater which was formed by a collapse and not a strike !! [Location - Crater Lake National Park, Oregon]

Road to Rendition [Location - North Lake Tahoe]

One below the other should be one and the same, maybe a few centuries from now ;)

[Lower & Upper Yosemite Falls]

Lower & Upper Yosemite Falls to the Left, Half Dome in the center, Vernal & Nevada Falls to the right [Location - Glacier Point, Yosemite National Park]

The bridge on the right is not photo-shopped !! [Location - Zion National Park]

Water being confused if its getting dark or if its gonna get brighter... [Location - Zion National Park]

A wedge shaped rock but worth climbing. [Location - Angel's Landing, Zion National Park]

Jeremy Clarkson style - "Some say, this is where angels used to land, or they still do!!" [Location - Angel's Landing, Zion National Park]

Valley view with a few sun rays peeping through. [Location :- Angel's Landing, Zion National Park]

"I'm Arnold. Get DOWWWNN !!" [Location - Angel's Landing, Zion National Park]

Just cant help myself from saying "Daya.. darwaza tod do" [Location :- Outskirts of Bryce Canyon National Park]

Earth showing its cocky attitude to the sky with its fingers [Location :- Bryce Canyon National Park]

 Fight between water and sandstone over time [Location :- Bryce Canyon National Park]

"I'm still Arnold. Get DOWWWNN" [Location :- Bryce Canyon National Park]