Joe Lewis's Blog

Thoughts, Musings & Stories

Your Pay Scale Has No Obligation to Grow With Your Skillset

I have close to 10 years of experience in the software industry and I guess I’m eligible enough to make this conclusion: Your pay scale has no obligation to grow, as you grow your skillset or professionalism.

Your payscale is heavily dependent on what corner of the business your role is playing a part. A Manager can make big and small decisions all day and still earn a lot more than a support staff working 60 hours every week to ensure customer happiness.

You could be the best Rust programmer in the company, but someone who’s able to do tiny tweaks to Salesforce to increase sales efficiency of the organization, tends to get paid a lot more than a heavily-skilled Rust developer in the company.

These are just random examples. The point is, if you want your career rewards to increase while you grow your skills, make sure you position yourself in roles that actually make an impact in the organization.

I wish someone had taught me this early in my career.

I Made a Chrome Extension to Stop My Browser Tabs Growing Indefinitely

I frequently found myself ending up with 50+ open tabs in my browser, full of stuff I want to read later. I wasn’t ready to send them to bookmarks, because I know what an endless abyss my bookmark list is. Once it goes there, it never comes back to me again :)

So, I made a chrome extension that allows me to mark and close tabs, that I want to read later. This extension remembers them and emails me every monday with a list of links I wanted to read. Neat, isn’t it.

If you want to use it for yourself, visit: https://closetab.email

As usual I’ve open sourced the code: https://github.com/joelewis/readmelater

The Type System of the Future

Strongly typed vs More-strongly typed

We know types help us eliminate certain classes of errors from programs. For instance, it’s impossible to write an executable program in Java that adds a string to a number. If you’re a big company already using Java, you’ve hypothetically saved some significant amount of \$\$\$ in terms of developer productivity. But we still lose \$\$\$ with the most expensive mistake ever made in the software industry - Null. I’m pretty sure while you are reading this, at least one programmer across the world is either running into a null access exception or debugging one!

Looks like Java is not a strongly typed language after all. It’s just less-weakly typed than python. Well, we have Swift or Kotlin, which when written in a certain way can be statically verified that the program is free of null errors. Or even better, Haskell – which eliminates possibilities of an invalid program state altogether, with its strict immutability.

What I’ve really found is that type-checking is an incremental concept. As in, for every strongly typed language, there’s going to be a stronger-typed language. The interesting question is, what will happen if we push this to its extreme. Can I write a program that is free of all classes of errors? Can there be a language that eliminates the possibility of making any mistakes in your program? Of course, it’d be a nightmare to even code in such a strict language but it nevertheless is an interesting question to ask!

What are Types really?

For the answer to be obvious let’s step back and dig a little deeper into what Types really are?

Surprisingly, Types didn’t originate from the computer science industry. It actually came from mathematics! The type system is just one of the many secret affairs computer science had with math.

Consider the phrase “I’m lying”. How will you communicate that intention to a computer or an alien? You convert the phrase to math. It’d become

“For any proposition p that I affirm, p is not true”

But wait, “I’m lying” is a proposition itself. If this proposition is p, then “I’m lying” is not true too. So now we’ve ended up with an error. If he claims he’s lying, the alien will understand everything he’s saying is true - which defeats the entire point.

Here’s another such error:

A barber in a village shaves all men who don’t shave themselves. Now the question is, who shaves the barber?

If the barber shaves himself then he comes under the category of people who shave themselves, so he shouldn’t shave himself. But if he doesn’t shave himself then he comes under “people who won’t shave themselves” and therefore should shave himself!

We’ve ended up with a paradox. Actually a pretty famous one at that. It’s called Russell’s paradox. Russell tried to base all of mathematics on top of logic and was troubled by this paradox.

He came up with something awesome. He went into Zen-like mode and said there simply cannot exist a predicate (I’m lying) that includes everything. There should be a class of results that hold true for a predicate and he called them, Types.

For example the “I’m lying” proposition holds true for all propositions I say, except that one. “The barber shaves all who don’t shave themselves” holds true for all men, except the barber. These are valid exceptions because they are of higher-order! (If the term higher-order rings a bell, it’s because this is how first-order logic and higher-order logic was born)

This is just like how when we write int sum (int, int); we say sum() is a function, but that function only holds true for inputs of type (int, int). See the analogy? This is where we stole types from.

The twin faces of Types

Remember our original question? Is it possible to write programs that won’t allow us to make any mistakes? We dug into the history of Types because as you can see what type means in math is (though similar) very much more than what we call datatypes in software. It implies we can do better than just data-types.

For a function f(x) = x*x that outputs the square of its input, current languages that we use allow us to express that the function takes in only integer values and not strings. i.e the data-type of arguments.

Is there a language with a type system powerful enough to express that the function adheres to the constraint: for every input the result will be the square of the input?

If such an arrangement exists, then it’s safe to assume that a language in which no erroneous programs will exist is possible. This is where the deeper meaning of Types in math, helps. While Russell proposed the Ramified theory of types to contain paradoxes, following his lead a new theory emerged - Proposition as types by Brouwer.

This theory says that any proposition has a type. The type of the proposition is the proof of the proposition. For example, consider the proposition: “Chennai is in India”

The proof is based on the below facts. Let’s call it, Proof-A.

Chennai is in Tamil Nadu. ( Chennai ⊆ TamilNadu )

Tamil Nadu is in India. ( Tamil Nadu ⊆ India )

So Chennai ⊆ India is true by transitivity.

Here the proposition “Chennai is in India” is of the type “Proof-A”. This proof part is interesting because this proof is nothing but some logic, that is in-fact closely related to a computer program. This is another interesting observation. It’s called the Curry-Howard isomorphism. Curry observed that the properties of a proof in Logic, is very similar to the properties of a computer program.

And that is again how math and computer science is married again!

The world of statically verified languages

So it is possible to statically verify that a function always returns the square of its input, it just needs a proof (aka its type). And as Curry observed, proofs are nothing but computer programs, except that they work with facts from the compile-time, instead of the runtime!

For example, you can write a proof that your sort() function, can never return an unsorted collection. And the compiler will run your verification at compile-time and guarantee that for you. Much like unit tests, but at compile-time.

As expected, there are programming languages out there, that use this Curry-Howard isomorphism to statically verify its correctness. Languages like CoQ, Agda, Idris. Folks use these languages to build mission-critical software for rockets and airplanes, but the rocket-sciency nature of these languages is stopping the general business world from adopting these languages.

I have a strong feeling that the industry will slowly drift towards languages like these that have a type system powerful enough to express the logic itself as statically-verifiable types. A world where every library I use is statically verified and proved to be correct will be an awesome world to live in!

Edits

  1. Grammatical mistakes

  2. Wrong links

Helpful Resources for Understanding Bits of Lambda Calculus, Functional Programming, Types and Monads

A cheat sheet for Set Theory. A basic understanding of Set Theory and its jargons (domains, ranges, functions) will be handy while reading most programming language theory-ish articles. Just understand Set Theory, Type Theory and the likes are just different ways of defining the same mathematical phenomenons.

http://people.umass.edu/partee/NZ_2006/Set%20Theory%20Basics.pdf

The Turing Machine - Alan’s original paper explained for mortals

http://index-of.co.uk/Theory-of-Computation/Charles_Petzold-Annotated_Turing-Wiley(2008).pdf.pdf)

Algebraic data types (touches on recursive types)

https://en.wikibooks.org/wiki/Introduction_to_Programming_Languages/Algebraic_Data_Types

An excellent resource for diving into Haskell. Abstracts away most math theory behind Haskell but you’ll get a hang of the language.

http://learnyouahaskell.com/recursion#hello-recursion Start with recursion, just for the fun of it!

Don’t let the title mislead you. I had an Aha moment on Lambda Calculus itself reading mid-way.

https://cdsmith.wordpress.com/2012/04/18/why-do-monads-matter/

YCombinator - decrypted for mortals.

https://lucasfcosta.com/2018/05/20/Y-The-Most-Beautiful-Idea-in-Computer-Science.html

The science of functional programming (Haven’t finished reading yet, but looks very approachable)

https://github.com/winitzki/sofp/blob/master/sofp-src/sofp.pdf

Lambda Calculus

https://brilliant.org/wiki/lambda-calculus/

Harvard CS121 Book for Theory of Computation - with math background explained.

https://introtcs.org/public/

8 Signs if Someone's Good at Something

Written in first person perspective for clarity. The list ranges from novice to expert in that order. Exceptions exist, and this is just a list of common symptoms, not a roadmap.

The thing tends to excite you a tad bit more than it does others.

Reading / learning about this thing amuses you.

After a while, you tend to see tiny details others usually ignore.

Creating something yourself related to that thing keeps you in the zone.

You have fun being in that zone.

You can anytime pick creating / learning this thing over a dinner hangout with friends.

You start appreciating what others have done in this field in a much deeper way. That is, you’ve come to see the true genius of someone’s work. Not because the rest of the community appreciates his / her work.

You create stuff, that folks in that field find interesting and useful.

Awesome Compiler Resources

Blogs

M

Implementing a Simple Compiler on 25 Lines of JavaScript

Developing Statically Typed Programming Language

Let’s Build An Interpreter

Let’s Build A Compiler

Let’s Build A Scheme Compiler

Compiler Basics - Series

Write Your Own Programming Language & Compiler In Python

Implementing a small programming language

Introduction To LLVM

Just Read These Two Papers To Build A Compiler

http://turbopascal.org/ (Pascal compiler internals)

http://journal.stuffwithstuff.com/2011/03/19/pratt-parsers-expression-parsing-made-easy/ (A quick intro to expression parsing, grammars, and stuff)

https://github.com/melling/ComputerLanguages/blob/master/compilers.md (HN Links)

Experience of folks who’ve tried building compilers

I Built a Lisp Compiler

https://blog.sigplan.org/2019/07/09/my-first-fifteen-compilers/

My First LLVM Compiler

Books

https://craftinginterpreters.com/

https://interpreterbook.com/ (paid book)

http://www.buildyourownlisp.com/contents

https://holub.com/compiler/

https://t3x.org/t3x/book.html

http://hjemmesider.diku.dk/\~torbenm/Basics/basics_lulu2.pdf - Compiler Basics

Courses

https://lagunita.stanford.edu/courses/Engineering/Compilers/Fall2014/about

https://www.dabeaz.com/compiler.html

Tools

https://t3x.org/t3x/book.html

Questions to Ask Before Joining a Startup

I can’t boil it down to 3 because there are many things you need to understand, some details can be gleaned via observation too.

So some of the top questions you should ask IMO are below:

Runway left?

When was the last raise, what was the amount and time to raise it? What is the makeup of the investors (Angel vs. VC etc)?

What is the next milestone on the fundraising side?

What is the biggest competitive advantage/product feature the company has?

What are the biggest issues facing product/development?

How do you see my role in helping the company?

How can I best help the company if I come aboard?

What is the biggest opportunity you feel the company has?

What is the biggest risk the company faces?

Who do I report to? – this trips some startups up but can highlight potential issues.

How many employees are there today? How many do you plan to hire in the next 3, 6, 12 months? How many employees have left since founding?

Equity and the vesting schedule details. Share class information as well. Current dilution, outstanding shares issued and reserved is nice to know too as well.

For what it is worth and this applies to any size company, not just startups, you should also ask about what agreements they want you to sign and get them as early into the process as possible without being unreasonable. e.g. outside work policy, NDA, non-compete, non-solicitation, compensation, equity, etc. Some of these they may not give you until you agree at least conditionally verbally, but there should be no reason you can’t see them once you do. Companies that hide these until your first day make me nervous, at the same time that doesn’t mean they are bad, just maybe been given bad advice or have a lack of understanding themselves.

This entire thread is gold. This is a direct rip off from the top most comment. Bookmarking it here for reference.

How to Install & Use ProseMirror in Your Project

Prosemirror is an incredibly well-written rich text library. But, lacks proper prose on how to set it up in a project for beginner / occasional JS developers. By the time you’ve finished reading this you’d learn:

+ How to install prosemirror into your JS codebase (that isn't using fancy modern tools like Webpack, Babel, etc currently).
+ Why the installation is difficult.
+ Some modern JS practices.

This piece is intended for non full-time JS developers who were familiar with Javascript in its jQuery and <script> tag and ES5 days. If you are already familiar with ES6, module-bundling and imports you’d find no problem dealing with Prosemirror anyway.


The conventional way of using an external library was to include the JS file in your index.html in a script tag. This will attach a global namespace/variable for that library which will expose the library’s APIs and methods.

For example: You’ll include <script src=”jquery.js”> in your index.html After that you’ll access all jQuery APIs via window.jQuery or window.$ namespace.

Good News: Things have changed now. Most libraries are now written as ES6/CommonJS Modules which means your global namespace remains pristine.

Bad News: Things have changed now. There are extra steps that you’d need to make before incorporating these libraries into your JS code.


I’m assuming you are either familiar with ES5 JS and / or have a legacy JS codebase. To use Prosemirror in your codebase, you’d have to find a way to export necessary Prosemirror APIs into the global namespace.

Basically, build a ProseMirror.js file that you can use like jquery.js


Create a folder & file structure as below:

1
2
3
4
5
6
7
prosemirror/
       /src
       /src/index.js
     /public
     /public/index.html
       package.json
       rollup.config.js

Since ProseMirror is written using ES6, and we want to use it with plain old ES5 Javascript, we need to transpile ProseMirror library along with its global APIs to ES5.

We are going to do that with Babel & Use Rollup as our module bundler.

Create a package.json file

1
cd  prosemirror

Put the below content into package.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
  "name": "setup-prosemirror",

  "dependencies": {
    "prosemirror-example-setup": "^1.0.1",
    "prosemirror-model": "^1.6.4",
    "prosemirror-schema-basic": "^1.0.0",
    "prosemirror-state": "^1.2.2",
    "prosemirror-view": "^1.6.8"
  },
  "devDependencies": {
    "@babel/core": "^7.2.2",
    "@babel/preset-env": "^7.2.3",
    "babel-loader": "^8.0.5",
    "rollup": "^1.1.0",
    "rollup-plugin-babel": "^4.3.1",
    "rollup-plugin-commonjs": "^9.2.0",
    "rollup-plugin-node-resolve": "^4.0.0"
  }
}

The package.json file declares all the dependencies to build a final ProseMirror.js file, including all the tools needed.

Now it’s time to use this package.json and install the dependencies. Run:

1
bash$ npm install

After the dependencies are installed, create a rollup.config.js file in the prosemirror directory and put below code in it. This tells Rollup to take our src file from src/index.js and spit out a ProseMirror.js into public/ProseMirror.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import babel from 'rollup-plugin-babel';
import resolve from 'rollup-plugin-node-resolve';
import commonJS from 'rollup-plugin-commonjs'


export default {
  input: './src/index.js',
  output: {
    format: 'iife',
    file: 'public/ProseMirror.js',
    name: 'ProseMirror'
  },
  plugins: [
      babel(),
      resolve(),
      commonJS({
        include: 'node_modules/**'
      })
    ],
};

The configuration is mostly self-explanatory. We are telling Rollup to first convert all fancy ES6 code to ES5, then resolve all dependency libraries from node_modules folder and spit out a final ProseMirror.js as a self executing module format.

Now’s the most important step – Writing the code to import all ProseMirror APIs and exporting them to window namespace.

Put below code into src/index.js

1
2
3
4
5
export { EditorState } from "prosemirror-state";
export { EditorView } from "prosemirror-view";
export { Schema, DOMParser, Node } from "prosemirror-model";
export { schema as basicSchema } from "prosemirror-schema-basic";
export { exampleSetup } from "prosemirror-example-setup";

Without Rollup we would have had to manually attach each API to window ourselves. Like:

1
2
window.ProseMirror = {}
window.ProseMirror.EditorState = EditorState;

But luckily we have a better way. Rollup can just parse this file and do that work us. Just run:

1
bash$ rollup -c

You will see a ProseMirror.js file being created at public/ProseMirror.js

All that is left is to use this compiled JS and initialize a ProseMirror editable area. We’ll include this js file in public/index.html and test out a small demo. Put the below code in public/index.html

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
<!DOCTYPE html> <meta charset="utf8" />
<html>
<head>
    <link rel="stylesheet" href="https://prosemirror.net/css/editor.css" />
</head>
<body>
<div id="editor" style="margin-bottom: 23px"></div>
<script src="index.js"></script>
<script>
// window.ProseMirror is active by now.
var plugins = ProseMirror.exampleSetup({ schema: ProseMirror.basicSchema });

var view = new ProseMirror.EditorView(document.querySelector("#editor"), {
    state: ProseMirror.EditorState.create({
        schema: ProseMirror.basicSchema,
        plugins: plugins
    })
});
</script>
</body>
</html>

Open up index.html in a browser and you should see a nice little rich text editor out there!

DISCLAIMER: However convenient this seems, this is a non-scalable way of using ProseMirror. For example anytime you want to use another API from ProseMirror, you’d have to edit src/index.jsto export that API for you. Also, polluting the global namespace is a thing of the past. Moving your project to a good module system (like ES6 Modules) would be more beneficial than the cost involved.

Spring Boot - How to Serve Static Files Without Reloading the Application

The default way of serving static files like JS & CSS via Spring Boot is to put your static folder under /src/main/resources/public. The Internet will give you all kinds of variations of this solution, but almost all of the solutions have a problem.

The static files will be bundled up into the JAR file (that is if you are using the embedded server that comes with Spring). This means you have to reboot the JAR / application everytime you change your CSS or JS. Things could get really painful when you already have a webpack compile process, that will be chained to a JAR compile process.
If you use IntelliJ, there’s a solution to auto-reload your app everytime static files change. This one has its own problems. If you don’t edit static files from your IntelliJ and instead use WebStorm / Sublime / VSCode you need to come back to the IntelliJ window once after editing, so that IntelliJ knows that it should reload static. Fail.

So here’s a solution that worked for me: simply put static resources out of the JAR and write a URL handler that serves static resources from the filesystem. This can be done by hijacking Spring’s default /static url handler and writing your own.

Put your static file in the root folder alongside /src.

1
2
3
4
5
6
7
8
9
myapplication/
           /bin
           /libs
           /src
           /static  ---------> your static folder
                  /css
                      /main.css
                  /js
                      /main.js

Include the following URL handler.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
/*
 * THIS ENDPOINT IS JUST FOR DEV.
 * PRODUCTION MODE SHOULD ALWAYS SERVE STATIC FILES FROM A SEPARATE SERVER
 *
 */
@GetMapping(value = "static/**")
public void serveStatic(HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException {

    /*
     * if /static/css/main.css lands here -> path = "/static/css/main.css"
     */
    String path = request.getRequestURI();

    /*
     * with IntelliJ cwd will by default point to "/home/.../application"
     */
    String cwd = Paths.get(".").toAbsolutePath().normalize().toString();
    File file = new File(cwd + path);

    /*
     * If you want you can have your static folder anywhere on your filesystem. Just make sure file object points to the correct path.
     */
    FileInputStream inStream = new FileInputStream(file);
    OutputStream outStream = response.getOutputStream();
    String mimetype = Files.probeContentType(file.toPath());
    response.setContentType(mimetype);

    byte[] buffer = new byte[4096];
    int bytesRead = -1;

    while ((bytesRead = inStream.read(buffer)) != -1) {
        outStream.write(buffer, 0, bytesRead);
    }

    inStream.close();
    outStream.close();
}

With this in place, your spring boot application doesn’t have to be reloaded everytime you change the font-color. Happy Coding!

Implement Decorators in Javascript ES5

There’s an amazing pattern in Python, called function decorators. It’s a set of declarations at the top of a function definition, that slightly enhances how the function behaves without actually polluting the function definition with related logic. I find it a really neat way to define a function and to re-use these enhancements/additional logic across functions.

If you are like me, in love with decorators and now in the wildness of Javascript, missing them – here’s a quick way to implement decorators in JS. My solution here, uses underscore’s compose method. But it’s easy to whip up your own version of compose, if you don’t really want to use underscorejs.

Here’s the short version:

Say, you have a couple of decorators.

One, which logs the return value of the function it decorates.

Two, which makes sure the function is called exactly and only once, no matter how many times it gets invoked.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
/* Decorator 1 */
var shouldLog = function(original_fn) {
  var decorated_fn = function() {
      var returnVal = original_fn.apply(this, arguments);
      // log the return value
      console.log(returnVal);
      return returnVal;
    }
    return decorated_fn;
}

/* Decorator 2 */
var runOnlyOnce = function(original_fn) {
  var isAlreadyRun = false;
  var decorated_fn = function() {
  // only execute if not already run
  if (!isAlreadyRun) {
          var returnVal = original_fn.apply(this, arguments);
          isAlreadyRun = true;
          return returnVal;
        }
    }
    return decorated_fn;
}

var myFunction = function() {
  // here goes your function definition
}

// now decorate it.
myFunction = _.compose(shouldLog, runOnlyOnce)(myFunction);

Or you can go for another version, if you aren’t really a fan of two liners.

1
2
3
var myLittleFunction = U.compose(shouldLog, runOnlyOnce)(function() {
  // here goes your function definition
});

All done! Now every time you call your function, it passes through the decorators around it.

The logic is pretty straightforward considering you are familiar with higher order functions.

1
2
3
4
5
6
7
8
9
10
11
12
myFunction  = f(x)

shouldLog = g(x) // higher order function where x = some function (f(x))

runOnlyOnce = h(x) // higher order function where x = some function (f(x))

chainedDecorator = _.compose(decoratorFunc1, decoratorFunc2) // h(g(x))

decoratedMyFunction = chainedDecorators(myFunc) // h(g(f(x)))

// to execute function with decorations
decoratedMyFunc(args);

Caution: Be cautious when you write a decorator that modifies the return value / parameters of the function. This could cause potential breakage of other decorators. Be careful when you decorate a method. Make sure decoratedMyFunction is properly bound to the intended context. E.g decoratedMyFunction.bind(MyClass)