In case of Leonardo's character he has a spinning top, that has a very peculiar property, when it's spun in a dream it never stops spinning, while obviously in the real "reality" due to the laws of physics it eventually stops spinning. This is sort of a "Deus Ex Machina" for the history but it plays really well and doesn't feel like one because the effect only becomes "supernatural" when its owner is dreaming, so in fact, the artifact doesn't have itself any magical property.

In terms of story though, if you squint, this is like a magical sword in a fantasy movie, or a special inhuman ability someone has in some other kind of story.

It's the interplay of that supposed property of the "token" with the "dream" universe of the movie that allows major plot progressions, and in a way lead, perhaps, to the most poignant question that the movie raises, when in the ending scene we see Leonardo's character in his supposed real home, we hear the voices of his children in the background, signifying they're playing outside, it has an idillic mountain/camp house ambience to the scene, yet at the same time, the spinning top is spinning on top of the table. The scene lingers a bit and cuts to the end. It's that lingering that makes it the most poignant scene in my opinion, because it kinda feels when you're watching just before it cuts that the spin top was already spinning a bit more than what would feel normal, and then it cuts and you don't have your answer. So without the spinning top you can't know if the character is eventually in "real" life and found the happiness with his family - that he continuously says is one of the motivations throughout the movie - or if the character is in a dream, addicted to a certain dream and is returning to it, and if so if either he's there voluntarily or he's stuck and doesn't know.

There's that final tension. I find some parallels to what our experiences in this reality, whatever it might be, can ultimately reveal us about reality itself.

In a way you can't really know if you're in a materialistic first reality, a conscious first reality, a simulation or a God made reality - that is functionally equivalent to a simulation, in the sense that for you, as the subjective experiencer of reality, you wouldn't be able to tell a simulation from a God made reality because they can look like exactly the same.

If I think about it, how could I know I'm in the "ultimate" reality, the real one and not in a dream, or a simulation (all those are functionally equivalent, so I'm going to refer to them as simulation from now on). I need either something internal or external to this reality to show me that, or I need to have noticed or understood some detail that would indicate so. Because how could you test it? And even if you were to be shown, or understood it by yoursel, or somehow woke up, got enlightened, whatever, then what? How could you know that you were not in another simulation or dream? And this is basically the problem. "We" don't have a magical spinning top and "we" don't know what "we" don't know.

So let's see the different types of philosophies that we can think of coming out of these different approaches we could take.

A materialistic first reality. This is what most atheists, marxists, utilitarians, etc, ascribe. They say there's nothing else than matter that happens to become conscious by chance and from there we get here through a sort of evolution. Notice here that when I say matter, because you are starting without consciouness, this can only be rocks, minerals, liquids? Not sure there can be liquids without life, but anyway it's this type of matter, not matter like tree bark, or flesh, because those can't exist without consciousness. This branch doesn't provide a framework for before, or after, sure, some theories might present but they're far from being even significant in gathered support for their views and not proved (some not provable with what we know and tools we have right now).

So the biggest gap in this philosophy, to me, is how do we go from unconscious matter to consciousness. I like to imagine it like this. Have you seen a lightning storm? One that goes for a while? Like those lightnings they occurr in that situation, in that context, atmospheric, temperature, etc, but they're fickle. The lightnings strike and disappear, the charge just dissipates into the material world. The conditions change, the lightning no longer strikes. So how could electricity be stored in matter in a way that would give rise to biological beings with consciousness?

It seems that a vessel that could contain the current in order to use it for consciousness has to be "designed" before (not strictly in the sense of intelligent design, just that it has to have come up somehow before) consciousness arises in the material world, because otherwise, how would it come to be? But then how would life get to that design without life a-priori, just from rocks and thunder? A rock charged with electricity wouldn't be able to give rise to intelligence RANDOMLY (yes, silicon indeed can be charged and "think", but it has to be excruciatingly designed at a scale beyond human sight, and it still doesn't evolve into a biological cell that has a self programmed will - I'm not even saying we can't get there, but I'm really doubtful that such sequence could happen randomly in a total material world).

The biggest problem is that without will/consciousness, that are immaterial properties, there's no reason for life to come into existance. Why wouldn't the universe be just rocks floating in space? Even if we assume that because it has no beginning and no end it's infinite, and by being so it guarantees that all possible combinations of matter will occurr at some point, and as such can be assumed to have occurred and we can use it as assumed condition, even if a complex structure by pure random chance, that would be 2.225074e-308 but we're assuming the universe is infinite so we can assume this will too happen sometime along the way to infinity, and as such we can assume that that point is in fact now (or like some billion years ago), and was able to hold electricity in a contained way, why would that lead to consciousness? It seems that SOME directed consciousness needs to be there prior to enable a proper vessel capable of using that energy for consciousness to emerge. A bit of a snake eating its tail, or as we say in Portuguese, pescadinha de rabo na boca (hake with the ass/tail in its mouth) situation.

That's my biggest gripe with it, it can't explain anything of the mistery. It also gave rise - until quantum physics started cementing themselves - to many branches of determinism. Like the thought that all humans are the same at birth (beyond the "rights"), just separated by class, hence material conditions, and that they're a white page that then can be culturally and symbolically programmed into an "higher" man - very common in all forms of socialism, be it national or international. Although it is clear that the cultural, material, social dimensions have a direct and significant impact on the adult human one becomes, it's not in that way. You can't make an assembly line. It doesn't mean that it can't provide philosophical advancements, or only has bad repercussions. As the saying goes, even a broken clock is right twice a day, and more than that, if you are using right and wrong to describe theories that means there's a point that should be the correct, which consequently also means that if you start deviating from that point for any reason, then usually the course to correct that will be to temporarly go in the wrong direction.

Variations, and they go back to the sperm and egg that eventually agree to fuse, result in individuals that have underlying "hardware" different between themselves. So while you can create a "program", your programming doesn't guarantee a deterministic result. For instance, imagine an overly harsh, militaristic, homophobic father. He makes it all clear all the times yet, sometimes, the son comes out gay. Or imagine a school, you teach a subject, the professor has an inclination, can't work around the bias, yet, even then, different people will react differently to the things he will be talking about, creating different interpretations. Now, in both cases, they all can exhibit and behave as if they weren't, and agreed, respectively, but their inner monologues can also be different from what they "signal" publicly.

Obviously, if used to an extensive extent and well done, you can almost "create a new man", the thing is that it never really changes, it just puts on a facade according to the programming. But still, eventually, these systems, even those that manage that, seem to falter and fall. It's the nature of the human that never really changes. With the advent of quantum physics the classical model of physics stopped being the only way to describe reality, in fact, it became obvious that it's not total in any way and only coincidentally very useful, so determinism suffered a blow, but it recouped by integrating quantum physics' randomness, and allowing free-will. Materialism on the other hand has a bigger problem in my opinion because how does it bridge that gap between the materiality described and the apparent immateriality of certain aspects of reality.

If we look at a conscious first reality, on the other hand, it's immediately out of reach from a pure physics description, since we can't even define consciousness in any relevant non-ambiguous form. A conscious first reality allows many more easier answers - because they can't be tested as of now, only theorised - such as, if consciousness is in fact the underlying basis of reality, solving how do we get to conscious entities is pretty straightforward logically. The consciousness "field" gives rise itself to vessels that can contain consciousness, pretty much as the seed of a tree knows what the fruit will be and how to get there. In the end, the fruit bears the seed right? As you see, it's pretty straightforward and in case consciousness is the basis of reality it would make intuitive sense that it was so that we came to be. Variations of this chain of thought are gaya theories, cosmic universalism, etc. Again, if consciousness is the basis then the universe evolution is a manifestation of the will of that abstract consciousness, and the coming into existence of Earth itself too, and the evolution of life, etc. Similar to gaya (Earth) theories, where it's all the same thing and interconnected system that manifests in heterogeneous looking realities while sharing the same universal basis. In my opinion, such a view is ultimately indistinguishable (to the experiencer) from a God made reality or a simulation, although its conceptual frameworks are different.

Now lets turn our attention to the next possibility, a God made reality. This branch of thinking obviously pressuposes the existence of a God that is above all. This is needed because how can you go from nothing to something? And if not, if everything, or something is always there, how can that be, how and why does it sustain itself eternally? Ĩf in your mind for some reason that second question leads to a nonsensical answer - the reason being you think there needs to always have a start and an end, and you don't believe (logic, personal experience, wtv) that the world is first and foremost material (because you had some experience that leads you to believe it's metaphysical due to syncronicities or wtv), then your only option is to believe in God, in some way. It's also why perhaps most of the initial advancements in science all started in religious contexts, it's very probable even that the experience of natural phenomena that seems to follow a logical reason strenghtens the belief in some logical order that precedes everything - it seems we can understand even parts of the invisible and all sorts of things beyond our natural abilities, and that those parts seem to be in their own way and context consistent (in the logical sense).

When we look at several scientists many do affirm a belief in God, and we can probably look at their unwavering dedication (sometimes ungratefully treated, not rewarded, etc) as the side effect of their belief in a logical God, because if God is logical then it must follow that reality, specially because he created it, is logical too, and if we're created in his image then the understanding of those logical connections should be in reach. At the same time, depending on how many people you would consider idiot, it could be used as an argument against the idea that God created us in his image, since there would be so many idiots that it would have to mean that God was in some fundamental level idiot too. But that is kind of elided by the argument of free will.

Going back to scientists. I'm not saying that the story of the apple falling is what gave Newton the idea for gravity, but I'm pretty sure we all have been, either directly or as observers, in situations where someone says "na, this must mean this, because of this and that", or "this isn't really possible in any reasonable way" and then someone goes "no, it's because this and that and you can see here and there", or "wait, but couldn't we do this and that, and then we get that? It seems like it should work". Right? People look at things, sometimes the same exact thing and they don't have the same ideas, associations, connections, feelings, being triggered. Taking the story of the apple falling, we could ask, how many people in the whole history of humanity have seen apples falling? So the question shouldn't be "If a tree falls in a forest and no one is around to hear it, does it make a sound?", it should be, "If a tree falls in front of you do you even see it?".

God has to somehow be bigger (in a energetic, intellect and informational sense) than the universe he's creating, otherwise how could he bring it to existence? More specifically, how could that entity be omniscient? If we know that each manifest bit of reality can't be reduced informationally to 0, otherwise it ceases to have any distinguishing feature from the remaining bits, and if it seems that the universe is infinite, for a conscious entity to be omniscient of every bit of it, it would need to be infinitesimally bigger than the universe it's omniscient about, otherwise where would the information upon which the omniscience is based exist? So we have to assume that if God created it, He has to exist beyond that which he created, and due to his other qualities, we must assume those other properties (e.g.: if God is omniscient about an infinite universe, he must be infinite Himself to accommodate all the quanta of information required to do any meaningful action that has any semblance of justice, morality and/or compassion).

It could also be that the things that matter are in fact invisible and in much less abundance and the exuberance of life is but mirrors, so that God would only need to keep track of a very small subset of reality to keep its supposed attributes of justice, fairness. Obviously, there could be other ways, but these are those that I can think of that make any sense. God could obviously be not definable or comprehensible by the combination of my own material body and mind.

The same with relation to other gods, if there can be many gods and they can each create "realms" totally independent from each other, then this is less of a "God made reality", an instead the direct equivalent of a simulation, since each "realm" (the combination of people believing and some sort of organisative structure) will have its own rules according to who created it or rules it. If that is so then it shouldn't be that we have a universal baseline of what we deem moral, just and fair, that has been trending to the same direction across the whole world, mostly due to dialetics, war and commerce. It should be that each reality of each God would be, to have any semblance of fairness, rewarding of the qualities those Gods desire. So multiple competing Gods doesn't seem to lead towards that universalism either, because you can't be fair if you accept different measures of moral, justice and fairness. That's why for a God made reality to make sense you need monotheism in my view. The hindus thread sort of a mixed way, but they get a bit lost on the definitions in my opinion.

“Whence this creation has arisen — perhaps it formed itself, or perhaps it did not. The One who looks down on it, in the highest heaven, only He knows — or perhaps He does not know.”

In the end they seem to not conclude anything about how or who created reality, which is a very sensible position but makes it so that it's not really believing in a God made reality, just not refusing that possibility, hence why I say it's sort of a mixed way.

In Christianity and other monotheistic religions they argument that God precedes reality, and it's God himself that designs reality to his will, including yourself.

Because this isn't a particular essay on each of these branches of ideas, but instead an attempt to showcase why they come to exist and how they mold and format our understanding, our ways of life, our choices, otherwise how could it fit in a blog post, there's probably entire libraries written about these topics, let's move to the simulation reality.

Like I mentioned before, as simulation I will treat both in the technical sense and the metaphorical sense, both the possibility of living in a software based simulation and that of living in an hardware based one, both are the same for our purposes because they would look exactly the same to you, unless you had some spinning top to help you discern (I guess in the hardware one not even that). The same with like, ideas of an alien seeding, or extraterrestrial meddling with earth bound creatures, spiritually, genetically or whatever, all those, render the same situation, for you, e.g.: if you were seeded by aliens but you think you have evolved naturally from apes, or that God created you, it would be equivalent to living in a simulation, although there's no intrinsic desire to "simulate" anything.

It's interesting to also think, as purely speculative paths, that if it was some sort of extra-terrestrial seeding or meddling, the end result, besides being technically a simulation, also poses those other questions that a pure software simulation hypothesis would pose, how do you know when it's the "ultimate" reality, the one that is not simulated? But instead in this case it is, how do you know who is the originary extraterrestrial "source", would they be the originals? Would they have also been seeded? And if we somehow came to the conclusion they're the originals how would that elucidate us about the nature of reality? How did they come to be, how did consciousness arise in them too?

That's an important point. In this chain of thought, given what we know about the human body, reality, mind, we can imagine a technical situation or context where we would be experiencing a "simulation". If we can have Neura links messing around directly with memories, if we can figure a way to manipulate those memories directly, we could create a totally simulated experience, that anybody under that experience wouldn't be able to tell apart from their everyday experience, at least while in it. Right? So one of the things that messes this chain of thought is the "why?". Because depending on the why, or even if you can find one for your view, changes completely the ramifications of your experience of the simulation. And without a why there's really no reason.

For instance in "The Matrix", the reason why is that the matrix uses human for storing energy, many people come out saying it doesn't make sense, energetically, to use humans as batteries. It's a "Deus Ex Machina" nonetheless, anyway, there's a whole bunch of ways of justifying machines needing full-body developed humans with a functioning mind, so that is not really a problem, a bit of suspension of disbelief is normal when watching a movie.

Why would a simulation even be created? Well, we could think of some reason, we already kinda do it in smaller scales, even with humans, we try forms of organisation for different institutions that are essentially simulations, even in our everyday life I'm sure you have also done it. Like if you ever tested a friend, a partner, a person, your child, on their true intents, or how they behave in a given situation or context, then we can, technically, call that a simulation. You've simulated an occurrence, or architected it, to see how something would play out. That's essentially a simulation, if the people participating in it don't know, it's even closer conceptually to the spirit of "reality as a simulation" that you can't be aware of.

So the first "why?" could be simply as a simulation to examine future behaviour? Imagine, an advanced species that has found a way to simulate their systems and uses it to test things and take decisions. For instance it might test if a certain kind of DNA leads to a type of person or another, and if it leads to the desired type of person, then they proceed to use that DNA fragment to generate the next generation of their own species, or to-be-parents choose from a pool of their own eggs/sperm possibilities and play it out in a simulation that runs in 3 secs. Lets assume for the sake of discussion that this makes sense. If this "civilization" was looking for people who were extremely agressive, because it's totalitary, utiliarian, military species, then perhaps the test is if you are courageous and ruthless enough to rise above anything and anyone. The further you do the better the DNA strip scoring.

On the other hand, if this "civilization" was looking for the type of person that is peaceful, looks to help others, doesn't mistreat others, then succeeding in that kind of "simulation", where the "why" is that, means you would end up something like a Christian, or a Buddhist, or some gurus of different traditions,- if you look at it from the point of view of the All-Knowing Father, the Eye in the Sky, Karma, they would be pretty interchangeable metaphorically with a simulation that is being run with an evaluatory purpose.

Why is this theory important to at least consider? Because it's sort of feasible. You could be in a simulation, there's plenty of whys and technically it looks definitively possible. Why is it important to not be consumed by it? Exactly because you could never know without external intervention if you were or not, even if this one you could be sure, the same problems will just repeat one level above, and then another, and then another.

If this is so this tells you you should not linger on it, since it's not up to you to discern. If it's a simulation all bets are off. You remember that apparition of whatever you had, and took you on a long journey through X and becoming Y. Well, it was simulated. You see? There's no possible escape. This doesn't mean though that you shouldn't be aware of its possibility.

Going back to our topic at hand, it's impossible to know without direct intervention if we are in a virtual, God-made, or natural reality, and if we have an intuition that it isn't random, then we are bound to two of those possibilities and we can't discern between them (without a noticeable crack in the reality that would be somehow discernible and that we would be able to understand, or without intervention of some external or magical entity/power/object).

There's some details that might indicate a virtual or simulated reality. The way reality behaves in terms of measurements would be consistent with a system that is trying to optimize space and processing power. The same for an almost infinite but almost empty universe. It can look vast beyond comprehension but computationally, because it's mostly empty with inert and unconscious matter that follows pretty simple simulations - simple in the sense that it probably would cost less power to process all the info in 99% of the universe, the inert part of it, than it would to calculate fully one human life. Perhaps.

The, what appears to be, fact that certain properties of reality can only be known one at time, in that we can't really measure those properties at the same time, or the probabilistic fields used to calculate things at microscopic scale - this would be appropriate in what we call, in programming, a lazy iteration, meaning, the values are not calculated until someone looks at them, or something that is either conscious or is in the conscious mind of some conscious being (perhaps only human, who knows), interacts with it in any relevant matter. When this happens the details unfold, but until then no additional energy had to be used to calculate/process any of those particular things at that level of detail because everything else can be approximated statistically from a known base.

It's only when consciousness interacts that there's this explosion of possibilities. Now, it's important to understand that this could also be a natural world and those "glitches" are in fact artifacts of our methodologies, tools and understading. Given that we know that nature doesn't seem to waste energy, and goes for the path of least resistance, we could imagine a natural world that assumes such a configuration because it's the most efficient, in whatever relevant manners of efficiency ultimately matter. In such a natural world consciousness could still be part of the fabric of reality, which would force matter to have "crevices" where consciousness could plug itself, making this natural reality compatible with a simu-God-looking reality and at the same time with most modern understanding of reality at that level, where in fact it appears that matter is less solid than thought and where it behaves as if influenced by non-matter factors.

Now let's change to more practical, mundane things. Lets say you live on the West, is the democracy you live under really a democracy? Or has it been captured at different levels, not even only national, by burocratic groups managing the allocation of public money? In fact, it all seems to be democratic. You have elections, numbers come, you can see tendencies, bias, clear favouring, undemocratic treatment, both institutional and non-institutional, but ultimately, when a shift starts to appear, it ends usually being able to fight through the institutionalised corruption and capture. Ultimately it's mostly an illusion too. If you were to think through, yes, indeed, depending on the country and context, individuals that are the front-face of governments, or governmental positions, many times aren't those who are really in control. Many are just the figures that need to go into the cutboard shapes.

They pass laws that they were told to, negotiate towards something with personal and ulterior motives, then hold positions in concertated spots that allow them to cover each other's asses, steal money directly from projects, funds, and budgets that should have gone to other things, get together in their best suits and dresses and commemorate democracy while all their countries can't even fuck to maintenance levels, highest consumers of most hard recreational drugs by several miles, alcohol probably too, same for antidepressants, where public services, infrastructure, governance, locally, become everyday more enshittified, but then the "overlay" institutions put on this makeup of professionalism.

It's covid, millions spent in shit, but you put your mask on didn't you little piggy? Even if it's fake you're still the piggy-mor, the real trampa, never forget when you look in the mirror. In 6 months or so there were already contagion contact apps, a few more, biometric vaccination passports going to be implemented in Europe. 30 years they still can't trace where millions end up, but for your uniquely associated government issued wallet, oh baby, it even has GPS real time tracking. All paid with your money thank you.

You see the motor of the economic growth was drinking pig oil, while paying champagne to themselves, so most evolved, so above the other ignoramus peoples, hogs, that managed to loose a leading market, where they had like 50 years of advance, being inside or having access to the richest markets in the world, tying the whole industry and those adjacent in weak international positions, while, not without a reason but still, calling others "PIIGS". The problem is the red crossings. It's easy to prop your whole economy when you're buying at a discount from an enemy that has just invaded a neighbouring country. And you keep it going for years. Squeaky clean.

Then look at the services, I'm not sure how it's in other countries but I hear and see examples that are in some cases even worse, I don't know if to laugh or cry. Civil servants, or public workers in socialist-tending economies (or others that manifest the same through nepotism, and such), because they're overly protected by the state (or nepotism, etc) have a for life tenure coming with it, for as longs as you just play the right game. You can't be fired for anything other than egregious faults for the most part or cabinet changes - but those are politicians, even though it mixes, here I'm talking mostly about the unchanging mass.

So people start treating those positions like they're something they're owed and that they own, like they somehow have the right to put their own emotions, opinions or will above the processes and law, transforming the position into their own little dick serfdoms.

For instance you might find out after you go get the previous contracts for your new rental's electricity and water that none of them match the real address registered in your contract. So you now have 3 addresses that don't match, the canonical one, the one from the property registration reflects an old address that no longer exists because the land was re-ordained and given new names and numbers, somehow though, because the budgets come but the outcomes not, there was no updating of the underlying records, who needs that, so the others have different addresses, besides the specific things different between each. It starts immediately here. This irregularity now seeps throughout everything it touches. It can create all sort of problematic situations that are the result of such systems. It's great to hide things too, because nothing is connectable, and it also means that if the State needs data to feed its decisions it's all a shithole.

Then you go, want to register a company, there's no calculator that can give you exactly what you're going to pay in taxes, and when, and how, no, you need to pay someone, to then not be able to tell you either, but you need to pay this person (the accountant) anyway. But the question here is, would you go to a restaurant, buy groceries in a store, or whatever, if that place told you, "know what? now you came inside you need to pick something, take it, and we'll tell you how much it is after and you can't return what you choose". Then add to this that you're forced by law to have this paperwork done, because there's no other possible way right. Really impossible to have a program that calculates immediately everything. Where would they steal from then?

Then they behave against the law themselves, in the regional counters, company counters, no responsibility, one says something, you waste one day fulfilling it, to then arrive at the other place, at your expense, and the person refuses to attend you, but accepts the money it would cost, and then you find out after a year that those things were never put forward because the change they should have triggered wasn't, and you only notice when you receive the yearly papers. No one responsible. This whore probably has worked there for 30 years, by the look of that diarrhea skin, going to work there until retired then retire with it all paid by others, the only reason she has a job is because probably sucked enough cocks or is the wife of some corrupt pig.

For you to understand the reality matrix you need to understand other things that aren't very pleasant. You know that saying, it's hard for a pig to understand he's a pig if his food comes from him not knowing he's a pig? You can put the cows smiling on the milk packets, tell your little children how it's all needed and for them while it's an excuse for you to keep being the same pig you always were. Your lies will catch up to you. Thank God.

You see in socialist rethoric, they always talk about how the struggle is in the name of the worker, how "work according to each one's capacity", and other pig lies. What they really mean by work according to each one's capacity is, we're going to be the genius bosses and you'll do whatever the fuck I want. You can see for instance the way the matrix operates in reality, through very subtle things. If you play the game of Civilization you'll see that one of the forms of government you can choose is "Communism". The game is a resource game, so you need to manage resources, and under "Communism" you get a whole lot of perks derived from the fairytale of "planned" economies. Your fields in the game start producing much more because the magic of communism automatically multiplies the outputs due to the genius level of organisation.

The reality is that the real genius communists in genius communist countries triggered several deadly hungers due to food shortages in a combination of retarded ideas, complete lack of organisational skills and systems coupled with endemic corruption. The genius in USSR Ukraine was developing out of the ass theories that were latter projected to have hampered significantly the agricultural output in several provinces of the USSR, one of the genius ideas was if and when the cultivars were rotting, to place them under the earth, as that would do some fairy tale magic thing. In reality as it was rotting, protecting the rotting thing from the cold and sun by covering it, with a layer of dirt, ended up speeding the rotting process manifold.

In China Mao manages to also kill a few and gave show of his enormous agricultural genius several times. One was the 4 Pests campaign, where he decided that to solve the problem of pests taking bites out of the perfect Communist productivity they needed to erradicate them completely, so they launched 4 campaings targeting rats, flies, mosquitoes and sparrows. The extermination of sparrows led to a several ecological imbalance that resulted, in the lowest numbers projected, in 15 million dead in 2 years of starvation, with some numbers going as high as 55 million.

This idiot was also the one who said some nice things about buddhism, the little pig, but now the CCP understood that Buddhism has more soft power than all their pig politburo together, and now is asking the Dalai Lama to re-incarnate in China. Imagine what kind of fucking retard you have to be. Then the real leap forward. After loans from USSR, help from other countries, a country of hundreds of millions at the fingertips and still these retards couldn't make it work. Then it worked when some Hong Kong chinese got the green light to build the first free economic zone of China, after the CCP had rejected it previously. But of course, the incompetence can only go so far because after a while heads will roll, so they went ahead and finally gave green light.

The rest is not history, because even the following growth was all dictated by economic partners that were capitalistic, with transfer of technology mostly coming from capitalist countries. China is now the country which produces more billionaires per year. And these guys do a parade for Mao. How do you want me to call you? Why would I want to be part of that or my society turned into something like that? Even faker than what it already is. So you can go on electric buses to your 9-5 to then eat a pre-cooked meal served by robots and keep your pig behaviours because now there's not even scarcity to contend for? Raising more little pigs that behave like individual cancer cells.

Then you look at Cuba, since 1958 under the Castros. Longer than many family royal interludes. But it's all for the people. Obviously that island is probably cursed because everywhere else on earth but communist countries, even in fucking siberia, people manage to create agriculture outputs in enough quantity to not need food stamps to distribute it. People living in houses that look like they were built on the first wave of Conquistadores. And the embargo doesn't count. For many of these years Cuba had direct MONETARY assitance from the USSR in EXCESS of the estimated losses due to the embargo - and this was done so that the socialist solidarity against the capitalist rotten individualism was in full blown display. So they didn't really have any embargo in the real sense of the word. The USSR had an almost as extensive area of influence as the USA and Cuba was receiving money in excess of what they would be able to produce naturally.

But what did they do with that money? No one knows. Ta ta ta ta ra ra ra ta ta ra ra.

In Bangladesh they had a few millions to fix a central area of Dhaka, the millions disappeared, not a single improvement done, people living worse than pets in the west, but then the fault is of the West, obviously, not of those burka pigs.

Once I went surfing in Bali a retarded local went out at the same time and shouted to me "Go back to your country". I'm fine if people don't want foreigneirs in their country. But this fuck head is there surfing because foreigners came to that island and set that style of life. What that retarded deserved is to go back 60 years and go work the rice fields for his life and then sell his little daughters to get fucked to make money to pass by. And in this case it's even more egregious, these pigs coming from indonesia, with their muslim pig habits, they go there and set bombs and kill not only foreigners but the true locals, want to impose their rules over others that had been there for millenia. Can you imagine what kind of people this is?

Go see what this beautiful people did to the people of Timor-Lest when they decided to keep their independence. Go see. Fucking pigs. And then they go around talking about others with their pig mouths still dripping shit.

Go see Venezuela, where quality of life was skyrocketing before and see what it turned into. An open whore house and drug production country. Almost 8 million people got out just under Maduro's rule. A whole ton of others live, again, worse than some people would allow their pets to live in the "West". But the fault is of the West, not their pig development plans. Being the drug lab for Europe and Spain more specifically. You never wondered? Why these countries manage to live from what is essentially a plant that mixed with diesel, and grinded goes for at least 50€ a gram in Europe. You never? And why amphetamines for instance there's no traffic into Europe? Because they have their own labs in the Netherlands.

This is why you never catch drug dealers. It's always some retarded looking fuck that lives with 2 girls, 2 guys, a dog, and a cat, in a 2 bedroom flat that is the great mind behind some hundred million a container pop trip crossing several jurisdictions. Of course you can't have cameras on the docks, what non-sense is that, or you know automated x-ray machines, public records, etc. In all streets except the politicians homes and institutions, a camera per sqm, where it matters, ooopsie, forgotsy.

In a way it's funny that it's the fact that they're so incompetent that got them caught in all sorts of surveillance apparatus. That's what you get when you're only worrying about filling your filthy mouth like a pig. The sad part of this is the ones that have a hold of that information might not be much cleaner.

Do you ever wonder why laws are made the way they are? For instance prostitution. Isn't it funny that all these feminist parties, and so-called "socio-democratic" societies, all illegalize prostitution? I would say that with exception of women born/married to powerful individual/families the only women throughout history to hold any real amount of power were whores.

But this no good for business. So they make it illegal for a women to decide if she wants to charge money for a blowjob. To protect them. How funny it's then to find out whore houses everywhere, attended by politicians, cops, the guy from the barbershop, everyone. Then countries like Sweden, so worried about the little ones and prostitution, that it's completely illegal, you can go to prison for it, and because it's a cunt society you just need one retarded cunt to go to the police to get you in a whole problema.

On the other hand illegal undocumented immigration? Couples with several undocummented children? All covered from toe to head, no documents, from a culture that has contracts to fuck little children. That's all good. We're very humanistic, philosophical diarrheas. Let's not do anything about illegal immigration, which is the single-most linked factor to children-trafficking sex networks. Hmmm, how interesting. They probably want to teach them how to play LEGO after some good swedish meatballs.

Then the pervasive lies. Every country does the same but I know some examples in first hand from Portugal. I came to the conclusion that the reason they call names such as "Dark Ages", "Fascism", and talk so little about them is so that the incompetence isn't more visible. In Portugal they renamed the "Salazar Bridge" (built during and in honor or Oliveira Salazar the prime-minister during the Estado Novo dictatorship) to "25th April Bridge" (in honor of the Carnation Revolution that ended Estado Novo). So people might think that the existance of that bridge has anything to do with the socialist/communist movement of the time and consequent socialist/commie governments. But it hasn't.

Then when you ask about why Portugal stagnated for so long and still is in the tail of Europe, they talk about the costs of modernising, that they implemented public education, and so on so on, that's why the moneys goes and the country doesn't. But when you look more closely - and I just want to leave on thing clear, I'm not in favour of a fear based dictatorship, fascist or not, even though Estado Novo wasn't fascist at all, and I don't think that "Futebol Fado e Fátima" is where I wanna be living either - during the "Estado Novo" they managed to raise the literacy levels to about 46/48%. This is almost 50%. We also have to take into account that the pivotal work is always the most difficult, because it involves setting up all the infrastructure, networks and systems that guarantee a system that can be in continuous operation throughout the years.

Not only that, they were involved in wars against enemies proxying the major global world powers across Africa, India and Asia. They held their ground pretty well and where betrayed by several international players all with their filthy interests, spreading propaganda about the plight of the african (which, obviously, wasn't completely uncalled for for Africa in general) brothers while helping in the destruction of countries that were at the time some of the highest quality of life countries in Africa, all so they could set up their oil operations and fight the "bad" portuguese that were enslaving the poor africans. Then after the so called revolution, civil wars, some that still are ongoing. I know there's at times interests behind both, but this just comes to prove that there's no natural "african unity" so far, even in the same countries, sometimes comunities sharing the same native language for centuries, they still fall to tribalism and in-fighting. So it can't all be fault of the westerner, independently if they did also hedious acts.

It's interesting that the country of the "Liberté, Fraternité et Egualité", as soon as Portugal found a profitable oil exploration in Angola started forming the first "freedom fighter" in Congo, that would later lead some of the first "rebels" against Portugal. They should add another one to their slogan, "et suiné". Trés high-couture.

And you can see the continuation of this on the narratives and how they're told. When people say "Portugal" was exploring these countries and took these countries, they're basically re-writing history - and I'm not saying that the Portuguese didn't do no wrong, pretty sure that is not the case - by framing it has if the "infrastructure" that existed in these countries had been stolen, or occupied by the colonizer, when in truth, they had all been planned, paid and erected by the Portuguese since around the 1500s. But when you tell the history in a certain way the picture you create in the mind of others is different. For instance, if you're black growing up in a western country, going through an education and propaganda system that perpetuates this kind of ideas without ever showing the counterpart, it's probably hard to not grow hateful.

You also see this for instance in many online indian comments, when the topics go around the "cast system", you'll see plenty of accounts making comments on how the cast system was introduced by the western elite, that before it was all peachy peachy. But history doesn't agree with this. I, by random chance, happened upon some information regarding a battle in India, between the Portuguese and an Indian General, where the Indians in an totally outlandish numeric advantage still managed to loose. As it seems, the general that lead the attack took that loss in such a way that he relinquished his duties and went to follow a monastic life. One interesting detail though, is that it seems that the portuguese had managed to get the help from some indian natives, that seemed to be "dalits". Meaning "untouchables". It's funny that the appearance of "dalits" in indian societies has probably to do with some dogmatic stance on meat eating, since that is considered unclean, and probably, because in many ways they agree on Ahimsa, non-violence, the only way to properly "show" them that their way of life wasn't welcomed was to turn them into dalits. Who knows right?

Another example is about the Inquisition, for instance in Goa. For all I could find, the Inquisition only acted on christians, it didn't have jurisdiction at all over non-christians. But, because the government was portuguese, and hence Catholic Christian, any person that was Catholic Christian would be in better light to improve its standing socially, economically, etc, as is natural, irrespective of what we might feel about it - there's probably certain positions that wouldn't be accessible unless you were a Catholic Christian. It's the same as when there's leftist governments and they fill positions with their allies. So this meant that many people would search for conversion, sometimes with only the objective of reaping the benefits of that status, but underneath would keep the same rituals, beliefs and ways of life.

I can believe that for instance in continental Portugal the inquisition was used, at least at times, to take things from people based on fake testimonies and so on. But in India it kinda doesn't make sense if you think about it. The portuguese had supremacy on the ocean, they won several battles completely outnumbered, made alliances with locals, passed laws to incentivise mixed marriages, etc. If they would want to take whatever they wanted (from lands they already controlled) they would have don't you think? Why would they do this half-assed dance to get to that? Bringing Inquisition to another continent, delegating part of the power to another institution, etc. But then I also don't know, it could be that in fact there were more abuses than cases like those, but it would seem that is not the case.

It's like when the marxistards attack Portugal, and the Spanish, and so on, and say things like, yeah they came and gave mirrors for lands, and blá blá blá. Again the same thing, these are the kind of things that pre-programmed retards spew. Things people who don't have a functioning brain say.

A mirror takes a civilization to be made. It's so unnatural that most animals take a long-while to understand, when they do, that its their own reflection. Of course, here you can say, they don't because they're not very bright, but you can, that nonetheless the Europeans knew the value of those things and as such were taking advantage. Ok. Now take a deep-breath and just think for a second, these same Europeans were buying fucking condiments with high quality grade weapons, mostly more advanced than what any of those local populations had access to.

Why would they trade condiments for weapons and ammunitions? The reason you shouldn't speak isn't because you're not entitled to a voice, it's because you're a fucking retard.

Another example. Many well-off women during slavery times would use slave milk-maids. This is always spun as an example of horrific treatment, by abusing a women for her milk to feed another woman's child. And while in part that can be viewed that way and I would agree that I don't want forced feeding, how do you concile a well-off women allowing their most cherished thing (at least most important even if not cherished, at that time offspring was an even bigger thing) to feed on the tits of a black woman and create an emotional attachment to another mother figure, that happens to be black. Don't you think there's something also a bit less ugly hiding in this act? Have you ever noticed how even across species, if you take care of a calf, a dog, a bird, a lion, or any other mammal, since they're newborns, that they will create an attachment to you for life. This would probably been known for centuries because animal husbandry has been a thing for quite a while.

So would you say that those children that would have been fed by black slave women would when growing up be less or more emotionally attunned to them, perhaps to their condition, and so on?

On the same topic, the miscigenation topic. Marxists will bring the fact that many paintings and cultural (propaganda at that time wasn't formalized but this would be similar) artifacts when dealing with the black african population in any positive light would always depict them as either being inserted on white households, being mixed race with whites, etc, not ever as example just by being black.

While this is in part true, and kinda can make sense to arguee that perhaps more only black lifting culture could have been produced, at the same time, you're again arguing that these evil people, that thought blacks were sub-human, closer to animals than to them, would somehow ignore all those ideas they had about them and go on and procreate, make families, and communities with them? It's that your thesis. That they were knowingly interbreeding with a lower species to produce handicapped offspring? Sounds really smart, I'll give you that. Not only that, but also, if the culture that arrived was "white", and they want to integrate the others, how would they properly depict that idea of integration other than the mixing of two different people into a mixed offspring?

But what gets me the most is the fake-revolutionaires. You know the ones that when they're in their democratic (they call fascist, colonizors, opressors, etc) countries they make all these noises, insult people, many times using lies either knowingly or unknowningly, as some of the examples already talked here, and nothing happens to them because guess what, for all its faults it's the best we managed without Kings of any kind that allows anyone to speak their mind, even when it's directly connected to their anus. But they could take their sorry asses, work one year anywhere eating ramen to put up enough money and then open their cellphone, book a trip, get a uber, enter a plane, fly directly to any country that they so like where they were explored so badly by the colonizers, get another uber, go to the place to check-in, all done through the cellphone, and go do manifestations right there where people are still trafficked and sold for dollars on a daily basis.

But they don't do that. Why? Because they would be killed and thrown into some road sideway and no-one would find them ever again. Yet they don't do that but go on public insulting, in all kinds of ways, people that lived 500 years ago, some who crossed oceans multiple times, spending months in the sea open to dying from the most varied causes, who setup schools in the middle of hellish places, who many times protected native populations from the greed of other colonizers AND OTHER NATIVES, in places where mosquitoes carry diseases capable of killing you while you sleep, sometimes doing the months long trip back to arguee in favour of the natives, sometimes getting killed, imprisoned or executed in these missions, yet these fuck tards that never did anything for anyone else than themselves (and in this I count family as themselves) think they have the right to come insult people that aren't here to defend themselves and with resources that amount to 0 in today's world changed completely the history of the planet while they repeat worn out slogans from their little propaganda books behind their keyboards.

Going back to the "propaganda". So when we look at the 46% literacy levels left by the Estado Novo against the current state we kinda see that they actually, in 50 years, did the same percent wise and starting from a base that was setup up earlier by someone else, those they criticize. If you then start thinking that in this process the access to education became paid, the taxes grew and the services detiorated, it starts looking a bit like incompetence. Then think, not only that but they basically had what amounts to "infinite" money coming from the EU.

So the "Estado Novo", not only was having 3 wars to fight (Portugal currently, as many other NATO members, weren't even keeping with the agreed 2% spenditure...) set up education systems, not only on continental Portugal but also to some extent in the Ultramar, increased the gold reserves in an absurd manner, a pillow that now these incompetent corrupt pigs are sitting in while keeping Portugal out of a world war and making bank while at it. If you start analysing these things you start to understand why the only thing that is talked about this time is the secret police, lack of liberties, persecutions, etc. They don't talk how they were betraying their country by fomenting the growth of USSR influence inside the country and ultramar territories. Because when you see the result of the communist doctrine (that to be truthful they couldn't entirely know at the time) today, and how the history went, you can't help but think that from the point of national security and truth, the secret police were right, as uncomfortable as that is to say and were in fact too lenient (not that if they had been less, they wouldn't have perhaps faced an uprising earlier, but ultimately, they were right but didn't manage to act on that with enough contudency).

So what do we have now? A society of doctors, where most people go to finish higher education at the University level but all put their covid masks, because the statistics for flu (there wasn't any possible covid test then...) in the flu season were so high, then you see the data, flu cases 0, covid X where X is roughly the same number of annual flu cases that suddenly dropped down to 0. Don't forget to not catch sun. So what is this reality I'm living in? Is it the short bus reality? A God joke? A random fart? What is it? The only way I would be so shameless while behaving in certain ways would be if I knew I was in a simulation, like some NPC where my own self being the equivalent of a metaphysical turd wouldn't matter, because it was what had to be done as an NPC. Maybe in Christian terms it would be the equivalent of saying a reality where the Devil is the lord and everything in it an illusion of his own making. It's weird isn't it.

It's funny how some weeks ago Maduro was cheering on Putin's Ukraine invasion and hoping for a "swift" takeover. Must suck to get ass fucked while you're taking a nap, but perhaps next time you don't wish for other people to loose their lives, freedom and right to self-determination to appease your pig friends. You're probably not feeling that smart now. Imagine being willing to drag your people into a war with someone capable to put you back into the neolithic in a single night of work, coming all fanfarre, waving his little sword around like some sort of brave heroic leader, so you can keep making your country poorer so you get richer.

Isn't it funny how the left always ends up tangled deeply with crime networks? It's almost a modus operandi. They disarm the population, relax the criminal penalties in name of tolerance, re-education, defund the police, invite foreigner agents to setup bizzybizzy, while always claiming for higher taxes, delivering shit. Have you noticed that Brazil for instance has some homicide rates that are almost at the level of war zones? Then you see these politicians all involved in drugs, prostitution, crime. Like the mafias this usually starts with at least a fair purpose, to give power to their community. But soon once power becomes established they start working against their own communities - not always directly but increasingly skimming and skimming and skimming and directly when someone dares to question. They loose the Robin Hood initial aura, that came with the protecting of the Italians or Irish, or Latinos, or the dispossessed, whatever, from the discrimination and attacks of other communities and get them respected, the lives improved but then, due to the greed that never ends, slowly metamorphosing into pigs.

And the left has decided to ally with criminals for the most part, in fact it's very in line with their philosophies, that power will not be exchanged freely by those who hold it, so it has to be taken by force. Isn't it funny that then they go on and immediately pass laws to disarm the populace? Fishy hummm.

Now I wonder, so these people believe, many really believe in this, to this day, that killing others, including innocent people through terrorist attacks, because they didn't want to share/give what was theirs in name of their abstract "soviet international" fairytale, was fair and necessary. A question of almost justice. Ok. So what would you think would be justice for people that have stolen that which is not even theirs? And in many times in magnitudes higher amounts than some of these other cases. Execution of them and their families in the public square? Execution of them, and any directly related family that is an adult and sterelization of the children?

I mean, if you compare the supposed "crime", I would say that that, in a scale that tries to be fair, would still not match what they did to so many people. This is not to say that the great industrialists/landowners of the time were some peachy smelling flowers, obviously, but I always find it funny how these people are always searching to divide others and then go on behaving like a retarded parrot cheering for their club without any thought given to the contradictions. Because if I was born at that time and in poverty as many did, I would have probably became communist/extreme-left leaning, because even if I did know it was ultimately wrong, as said before, to correct a wrong route, you sometimes have to sail in the wrong direction for a while. But today? With all we know?

Have you ever seen communists or socialists that dodge paying taxes? Have you ever thought how our modern countries work in economical terms?

You see they keep expanding the public entourage across all board, tied up with their man made constitutions treated as if they were God given and immutable, which is funny for people that are always pointing their fingers to religion (except the Muslim pedophile religion) mocking them for believing a "fairy tale" document, but then they do similar, has if it was given by god, when you think about it, how does it make sense? Aren't you doing exactly the same, except for you it's this particular document that is treated as if God given, because it goes with your club?

Lets do an exercise. Lets say we're in Narnia, and Narnia's current available state budget is 10.000€ and Narnia's private economy is worth 10.000€ too. I live in Narnia too, I open a business selling my services to clients outside of Narnia, after paying for the priviledge and being treated like some sort of criminal by the Narnia retarded public officers. Everybody in Narnia is very smarty-pants. They created a system that you need to open a LTD responsability company, with a society by quotas but single ownership, yet you have to do all the rituals as if you had a quorum to respect, have to hire an accountant to basically submit the papers that they already have or can be submitted online, that can't give you informations about anything. Tells you to go do the registering of the company on a "friends" office, where they charged 280€, on top of the regular 200€ you have to pay the state, to submit an online form, don't even have prepared a small binder with some instructions, information, nothing, they leave the contacts names in the official systems with their own, no follow up, nothing. 280€ for half an hour and then go whining about digital nomads and how much they make... Oh, almost forgot, and when I visited, before returning with the papers and everything ready, they had told me 200€, but then it was after all 280€ - since that was tax they "thought I knew". This is the kind of people that live in Narnia, always behaving as if their pig behaviours are fault of others and not themselves.

So I opened the business in Narnia, the state received 200€, plus 80€ of the taxes from the office, the % of the accountant, the office got 200€ too.

The state now has 10.280€, the economy grew to 10.200€, so on... Now I pay my first salary. As I said I work as a freelancer, now through a company and I had chosen a 2.000€ salary. My rental alone, plus internet, water, electricity, gas, phone, etc, would easily reach 800€ and more during winter - in a house where another pig from Narnia faked an energetic certificate and then when putting all documents in the contract binder, oopsie, just slipped it under the bag thinking I wasn't seeing and forgot to include the forged A category energetic certificate, because well, if that house is even a D would be too much, from raining inside, non-isolated ceiling windows with 3 finger gaps on the attic, etc, but that's another story. From that salary, because this is a company, in my name, with myself as a worker, the company pays out of its pocket (mine) the base salary, plus alimentation stipend that is legally required, one 12th of both the vacation and christmas subsidies, paid by the company each month or in total at certain dates (in this case I divided it to monthly payments).

So basically we're recycling the money through the tax system so they can skim it as many times as possible. They skim a bit in every one of these payment "steps" - from a nominal contract of 2.000€ I, "the worker", received 1.826€. But the company spent 3265€ (2450€ and an additional 815€ in Social Security) for the worker. Notice that I as an employer don't get any benefits from this, even though it's the employer who's bringing all this money from OUTSIDE the country and paying these things.

These numbers are all intentionally hidden, so that you don't understand how much the company actually pays for your work, so that they then can say "oh, they only pay 850€" or whatever when in truth is they pay much higher than that. Just take this example, the nominal contract value is 2000€ base salary, but in total the company spent 3200€ and the worker got 1800€. Adjust this to the reality of Narnia.

So after I paid from my own money, that was paid by someone outside Narnia, we can make the accounting again.

So now the state received close to 1.400€ in taxes, from this single payment monthly. So now the state has 11.680€, the economy has 12.000. This is so because in this case the money come from outside Narnia so both parts of the economy grow. Notice that the only benefits I get for this are as the "worker", I as the "entreprenour" that brings this businness I don't get any social security, or anything.

If you now take a public servant, that makes 2000€, the same nominal contract, how do you think we get the money needed to cover the total costs of 3200€? Let's see.

The public servant gets paid 1826€ just like me. So if the state has 10.000 - 1.826 = 8.174€. The private economy now has 11.826€ (the employee received the money, so now that money is part of the private pool). But the state employee also has to get his social security, subsidies, deslocation stipend, holidays, etc, all of that covered, after all, not only that, this person when retires will continue to be paid from the public treasury. So the state also had to cover the remaining 1.400€, so the state actually has 6.774€. And obviously the state entered a "contract" with this employee that guarantees after some years that they will have social security (unemployment, retirement, etc), that's why we're removing that value from the state pool, because those are extra outputs that need to be covered when the time comes (obviously, they'll be used to cover current expenses, due to how budgeting works, it's always behind, which is great because you can steal and only down the line does it break, when you're already writing books about the great public work and being paid to give talks).

So the difference when I pay, from having received money from outside, is that both economies, private and state, grow, while when government employees get paid, they get paid with the money from the state. So we could say that the objective of a state is to properly manage taxation first and foremost, since it's with that that then it can create a sustainable army, infrastructure, education systems. But when you look at the ways, the processes these states have built you can see they're completely incompetent, right? They don't even manage to properly tax their people because they create these pig kafka systems where they can continuously steal and hide.

But it's not only that, if I now buy anything I pay IVA for it, and that IVA goes directly to the state, the business gets my money, which again, didn't even exist in the economy before, because this money comes from OUTSIDE. When this person that gets paid from the public treasury goes to pay, obviously also pays IVA, but when you look, that money came in the first place from where it's going back to. So there's only changing of hands. Obviously, most public workers will be in families with people doing other work and so incomes will mix and come from different places, yet, it's pretty easy to understand what I'm saying here - if you can't understand it you really shouldn't be in any position of governance.

So if I'm bringing in 5.000€ per month from the outside, if 10% of the active population of Narnia did that, it would mean Narnia (not the state alone, but assuming the people would need to spend money to live there, and store it in local banks, which is the truth) would be receiving per year (50.000 people X 5.000€ X 12 months) 3.000.000.000€, from outside the country alone directly into their economy, both state and to Narnia banks that hold that money, while exporting basically services. But the retards can't do maths so this is what you get.

Now take into account that many of the people in these positions act like they're the last cookie on the packet, because you have to go through them to get anything done. All of this while being paid their whole fucking life with the money of others. Obviously, I don't want to work for free, neither do I want others to do that, but if your income is solely tied to the contributions of others you better know your place you little retard.

How do you solve this? With transparency built into the systems, automated processes, some sort of limitation on contracts for public placements. I mean, why would someone be expected to work all their life in a guaranteed position, other than when involved with secret/security services/clearances? This removes accountability, the desire to change, because letting things stay as they are is the surest way to guarantee continued employment, not rocking the boat, even if you're seeing unlawful or plain criminal actions - and in a way you can understand why it is so, if you have children to feed, house to pay - so the system has to be built to not suffer from it where it shouldn't.

I should be able to do ANY action online, and then, for a whole class of actions, I would need to go to any public service and do a quick input of the action number, on a biometric machine, that would then just print a receipt of confirmation. So you could have all the basic actions be automatic just online and anything more important guarded behind a simple two-step process that doesn't require any other input from any other person.

So you do the action online in the morning, then during your lunch break you go to validate it if needed. And now you have video-proof of the individual (because the public service has video surveillance) going to a public service, then entering the "confirmation" code along with their biometric details to confirm, and getting a physical printed receipt, the state has the digital records and also the physical original/duplicate emitted to the person.

This means it's not only or all "digital" and as such no longer problematic from a security and data integrity, tampering, hacking, etc, point of view, because important actions will need you to physically go to a place, solve a biometric challenge and confirm your intent, while producing along the way a bunch of auditable artifacts without needing ANYONE ELSE to waste time dealing with it.

The same with transparency. Taxes, payments, subsidies, should be calculated automatically from the minute you're interested in opening a company/getting a job and you fill a form putting employees salaries, expected revenue, etc. And this would be VALID legally unless you did something illegal or you didn't include the proper items to simulate. Otherwise, if what you input is the truthfull information you should know up to the cent each month, each payment, each tax return, you will get with exact dates.

This should all be automated, because it's exactly in all these manual processes and shit systems that the corruption hides. All under a cover of pretending to be small minded and a little retarded to not raise suspicions. There's other things that are in dire need of happening. Public servants in any condition with exception of special forces/secret services have to have a completely transparent financial life. All public institutions have to provide publicly and guarantee its ACCESSIBILITY all their spenditures and who signed them. All benefits, bonuses, social security payments need a confirmation period of at least 8 years for any public (state) company, public servant, politician, etc. So that there's time to both access the repercussions of the work/spenditure and if any illegality surfaces.

This removes all the shit-layer of parasites and sanitises the pigsty. But the pigs are already squeaking and putting their fingers where they shouldn't.

Once they come with vacination passports and integrated bio-medic chips, you'll need to force, even if by way of civil war, that it all starts by implanting those in the public servants, politicians and friends and make sure the system is publicly disclosed and auditable. You don't need to play this game. They have no power but they want to convince you they have. You don't need any of these things. If they made law be respected, criminals swiftly dealt with, proper immigration laws, you wouldn't have all these problems they want to solve with these solutions.

It's like forcing age-restriction on internet access, the way they projected is just for control. My first porn movie was a friend who found it in someone's secret stash. I would have been at most 11 or 12 I guess... It was a VHS tape, called "Dog luck" (or something translated to that in english), it was a German movie, where two girls take turns getting fucked by a german shepard dog under the caring eye of a male adult who just talks through the actions. I didn't need internet for this. And this wasn't the first and only VHS we had access to, a girl once found one and we did a cinema session in her house, about 12 people, to watch it, this time only humans were in the movie. Except one guy and one girl at that time, we the boys were really cuckolds, and didn't take any action... All this to say, that what this creates is a black market, the same way we got older people to buy us cigarettes, alcohol, you'll have kids trying to get sexual content access - intermediated by adults, is this better? I think the effort should be in making OS/hardware that are safer by default (not locked down, just not purposefully written to be full of holes), but even if that doesn't happen, parents already have ways of preventing access.

The same with some of the internet related rulings, the little pigs in the European Parliament pass these laws where they fine the big players but then everything just stays the same, no real change, just extorsion. Then they want respect. Imagine this, you want to build some business say like Paypal, Uber, AirB&B, in Europe... How do you think that would go in terms of regulamentation? Would never get off the ground, because all those other interests would not allow it - taxi associates have to pay for certification, etc, same with rentals, etc. These platforms all skip that substituting it with their own selection processes. So what happens? European businesses can't go these ways and try these models but then Europe goes and says, you know what, they, they can operate under these models, it's just you that can't. Then they have the grant mafia, where there's all this innovation grants and it's always the same guys getting them, a burocratic paradise where many leech off, because you need a whole bunch of burocratic tokens that require some know-how and inner knowledge to make it go through. Are all projects crap? Obviously not right, but damn, it would be much proper to diminish this and remove costs for creating enterprises, from micro to whatever.

Look at the European Union, all these years and not a single unified way of making business across countries and for the external world for small/self-owned companies. Everything the same as if you were doing business locally, same taxes even though you're acting as an exporter for services, etc. It was more important to be throwing moustache parties.

Then instead of removing taxes and impediments to creating businesses they just make it worse, individually and at the European level. And my problem with this is not that I don't think something like an European Union doesn't make sense, but what they set up is basically this system to skim money without any accountability, delegating decisions behind doors to unelected interests. The grant systems, loans, funds, little mafia run groups taking turns grabbing funds. Instead of this crap, wouldn't it be easier to just remove all costs to create, manage and close a company?

That would be something huh? But then where would you do your "socially worried redistribution of wealth" to your friends pockets?

You would have 2 free company registration, maintenance, wtv per 5 years. If any or both of those companies produced 1 year of employment for at least yourself with taxes paid, you would get again free company registration, maintenance, etc. Just plain, straightforward systems. Then if you want to audit, guess what, easy. The person building something, even if its own income and covering his supposed "social costs" to the state, shouldn't be paying this shit.

So people that go around jumping from job 6 months, another 6 months, then unemployment benefits, then another job, then unemployment, they get these benefits, and those that are at least covering their asses are extorted. At least remove the costs idiots.

Now lets talk about on-line identities. I projected a platform a long-time ago, dvouch, that would allow digital validation of identities without leaking anything at all, if the user so wished. The way it worked is the user registers on dvouch, then chooses from a preset of vetted platforms, and dvouch would connect on behalf of the user to those other platforms and extract information from them. Things like upwork.com, facebook.com, youtube.com, image selling platforms, etc. The idea was to expand it to all relevant platforms in use as dvouch grew. Anything where you had an identity and public information (or accessible behind the granted access). What this would allow is that for instance, I could use any alias for my name publicly, for my dvouch profile, then connect to my real upwork profile, by showing proof of credentials usually, extract the job history from upwork, and now display it in several manners. If I want anonymity I would choose to show that data as simple bucketed graphs - number of jobs on dates, success rate, total feedback, without showing any particular details that could identify me or those who hired me.

If on the other hand I wanted my profile to be public and publicize those same jobs I have completed, I could instead show not only the aggregate graphs but also a list of all jobs, with their rates, feedback, etc.

Because only a single account can be connected to each of these external providers and because it's dvouch that handles the data and displays it, the user can customise a lot of its profile but not the data points, those are always based on data coming from the real apis dvouch interacts with, anyone seeing that profile can know that whoever that is, those jobs happened, the stats are at least real. Then the profile had a simple contact form section where you could interact with the user by sending a message. No e-mail was disclosed, but allowed you to verify that by getting an answer it was really who the person said he was, as in, if someone sent you the profile and said, this is me, check my work history, you could go there and then send a direct message to the owner of that profile and establish if it's the same person that's talking with you.

Of course, without any way of really verifying the person who created the profile and connected the other platforms (no state/global identity system), it could be someone else, but since only one same account from each of the other platforms can be connected to a single dvouch account, there's not many ways to take advantage of it, and is not much different than someone hiring someone to do the take-aways interview, phone-call or whatever and faking some documents, it will always happen, but in this case it will at most happen to 1 of each external profiles per account.

I do believe that states will have at some point to create such systems (publicly) and that in part, as it already happens, surveillance will be built-in. Still, a citizen should be able to yes, claim they are someone online and prove it, through some secure government provided platform (auditable), because this is useful, but at the same time should be able to go anonymously online, and up to a certain extent not be tracked by 3rd parties and by the state unless in a grave suspicion (in the case of the state it's not even that the records can't be there, they theoretically could, but they shouldn't be accessible in any way unless under strict but reasonable conditions, and that's the hard part). It should also be possible to opt simply out of this system by not being forced to participate through it, but I imagine that even due to simple commodity and the advantage of having more data, that it will be impossible to not be registered somehow, as is already the case today, except it will be accessible almost real-time digitally instead of having to be dug for in real-life book keeping (while still being kept in that form as well somehow).

This also automatically separates things for those using the on-line spaces. If you're saying something outrageous but you're using a non-verified account to post it, I can approach it with a bit more caution from the get go, be the idea, the content, etc. If you use your verified name, your real person, exposing yourself, then even if I don't agree with you, I can know it's you. I can perhaps understand better some things. The problem with this is that it would sink the current web-ad model. Nonetheless, I don't see any reason why ads can't be made to work in other ways as well.

Then you have minorities and the discourse of the left again, all corrupt many times in bed with these criminal organisations. For instance in Portugal there as always been a problem with gypsies. Ever since I remember, when growing up an older boy, Luís (I can't recall if he was 14 or 15 or something like that, he was not like friend directly because he was older, but he lived closed by one of my best friends and was friend with that friend's older brother, who was his age), got stabbed to death by a gypsy, on a local fair, like animal/agricultural fair. Gypsies that traveled nomadically would visit fairs and sell their wares and this time although I don't recall exactly what happened I'm pretty sure it was not warranted a death, because most probably he was telling them something he was right about. Nothing happened. No one could find the gypsies nor who did it. A kid just lost is life like that.

But there was never, not even from those who knew the kid, any direct antagonism towards "gypsies" simply because they came from gypsy families. There were at least 2 gypsy families that had settled in that city, one had a big store, big window panes, right at the entrance in one of the nicer avenues of the city, the other a bar of dubious nightly activities (and there was at least a more famous "night" bar in town owned by others). The textile/clothing owners no-one mentioned them as gypsies, unless to disambiguate, they were no longer gypsies although they still had a bit of style, flavour if so to say, maybe 1st or 2nd generation to have settled, super nice people from all the times I interacted with them. The other family one of his sons was a very good friend of mine and of our other mutual friends, even though the bar was shady, our parents had no problem in letting us sleep over in his house and he did sleep over mine and others in our normal group as anyone else. No one even thought of calling them gypsies, much less gypsies in any negative way and one or two generations before they would have been living the way that the term gypsy usually implies.

Later on in another city, more to the interior, I was working on a coffe/restaurant and there was also seasons in the year where gypsies would come to visit. In that place the owner wouldn't allow us workers to serve gypsies before paying after too many fights involving unwillingness to pay, threats and so on, he decided no more that year. I asked why and I was told that besides those issues he himself had in the place, there were always robberies while they were camped around the city. Every year. I never thought about it anymore, later also went to university so only returned in the weekends of my first university year to wait weddings and, in one of those weekends, I was told that a gypsy encampment near that city was set on fire. They never settled again there and all those issues seemed to suddenly disappear. There's also other nearby cities that have some issues with them but there they live in government provided habitation, they're not part of these nomadic groups.

So there's always been quite a bit of tension. At some point I lived in Peniche, where there's also a notorious community living nearby. I knew a few people living there, some doing uber and bolt, many people complained, refused to serve the area where they lived for deliveries or pickups, due to multiple issues, or fear of issues. One day I had to go to the hospital, I went very early. I arrived not sure if around 8 or 9 am, there was big gypsy family sitting on the waiting room. A mother with many children below age 6 and one or two older than that. Immediately after sitting one of the boys come talk, ok, I strike conversation, it's a kid right, 4 or 5 years old, 3 phrases in he's doing gun signs and "pum pum" sounds and telling me his father has guns. Cool. I cut him off. Remember, owning guns in Portugal is illegal unless you have a license, some rifles can be owned if you have an hunting license. I sincerely doubt any has a license obviously... After some 15 minutes I start hearing the sound of water running. One of the little girls, sitting by the side of her mother, both on those waiting plastic chairs, was on her knees on top of it pissing on the chair, dripping to the floor. And the mother nothing. Just like it was not her problem.

5 minutes after the father comes out, sitting on an hospital wheelchair, surrounded by a doctor, a nurse, with serum being drip-fed, all good. The mother picks their children and just goes away and leaves the chair in a puddle of piss dripping to the floor, not a single word.

I don't know but, honestly, my parents always owned dogs and cats throughout me growing up, and I don't ever recall a dog or a cat not learning to not piss or shit inside houses, not even only our house, just houses in general in less than a month. Somehow these people, can't teach 4 or 5 years old these things. So they're in an hospital paid by portuguese people (they're the ones who don't want to be portuguese), raised by the Portuguese Republic, offering them a service they don't even pay, nor their community can provide themselves, as they don't pay other things (these types of gypsies, not gypsies by blood that have decided to just be portuguese and made their lifes, these "true" gypsies many live in government provided habitations where they pay a symbolic rent, sometimes 5€, and get benefits for every child, which they then can marry as young as they please between them because of culture and keep doing this).

There was at least 1 mother with a little child there, a bit far away from me, I was in the middle with a bunch of chairs between, looking like she just woke up at 6 am to bring the child, had to miss work, also known as "looking like shit", and some other people there too. And then they're sitting on a public hospital, with the known conditions and already known problems, and they still have to put up with more this shit, and if they say something they'll have problems, might be threatned or assaulted (my daddy has gun bum bum) and police, public officers do nothing. So the people that do everything as required in their duties as citizens are the ones that have to put up with this sort of crap caused by others who refuse to be part of that society in any helpful way. The same for the guys threatning people in public transport, taking over public spaces like they're theirs, what the fuck are we doing over there?

So instead of animals learning how to behave, it's the others that have to live like animals to not offend them. And the left is playing a problematic game, because this is the continent of Hitler and many others, and then when it has no longer a proper non-violent solution you'll look away and pretend it's not your fault in the way you've lead and modeled society. Obviously, this is also because in part you have allies in these criminal communities.

And this is the problem and mark of moral bankrupcy of the left. Without even talking about Islam and other things. They prefer to protect the poor guy who just happened to have robbed, did armed threats to others, sometimes exerted violence and even killed others, for the sake of their psychological being, at the expense of the physical, financial and psychological health of all others.

The same for immigration, oh, you want to become nationality X? Well, how nice of you. We're ready to welcome you, you just need 15 years of a clean sheet, no criminal charges plus either with 70% of that time under declared working contracts with private companies or generating a tax return of > than X. Complete fluency in the language. Etc. If you really want to move to another country you won't be doing shit, sorry. Yes, everyone (most) is entitled to a second chance but sometimes you screw too badly too. When I go on vacation I make an effort to not disrespect those that are hosting me, even if I diametrically think the opposite of them, although nowadays that doesn't happen because I just refuse to go where I don't agree with the general culture, but when I was younger I didn't have neither the same knowledge nor the same views but yet I still thought it was essential to not disrespect the countries I'm visiting even though I would also be paying tourist taxes, I didn't treat it as an entitlement, because honestly my individual contribution is in itself insignificant. To then see people that come and behave that way? What the fuck are we on over there?

So to me is a big surprise when people move to these countries with the pretext of asylum or looking for better life, or working, studying and then they get involved in drug dealing, crime, robberies, gangs, organised violence, human trafficking and/or are unable to let go of their "cultural" habits. If you want to keep them and they conflict with those who are already there, then go back to where you came from.

And it's not that those things don't happen with natives, but it's different when someone comes to another country and behaves that way, it tells you all you have to know about the mentality of that individual/group and you're not bound at all to please, host, or protect them, other than protect those that are law abiding and really intent on becoming citizings, as you would any future citizen. It's important to not let racist discourse take over because while it can be tempting there's cases where people of those cultures, race (?), language, as I spoke before, that were and are completely integrated so it's an awful thing, not even awful, it would be outrageous, to allow those to suffer because of those others.

And this is the last point. The actions of these so-called communities and their own ignoring of these problems hurts their own community but they don't care, they just want to keep behaving as animals and throwing blame on someone else.

I'll give you an example with something else entirely. Once, a couple years ago, I was watching a youtube video roundtable with some westerners and some vietnamese, and one of the vietnamese guys in the middle of talking about how vietnamese people are regarding foreigners just goes, yeah except black people, not really a racist thing, it's just we don't want that kind of behaviour here. This was around the time two afro-american (not american-africans, or amerindafricans) girls had made a scene in a nail/hairdresser shop in Bali that made the rounds in youtube and got them in serious trouble, they walked into a shop, got all the services, then refused to pay and then started threatening the shop girls and if I remember correctly started a physical altercation.

So now a black guy that never did shit to no one is just thought of as sort of an animal because of the behaviour of some.

And this is not like white foreign people aren't getting into issues in these countries regularly too, but usually it's in night out shops, bars, with alcohol, nightlife, it's a different thing than out of nowhere on a sunday morning a violent confrontation after applying nail gel or dread locks, that's why on those venues there's security, extra policing etc, because even locals will get into night life related altercations regularly.

Ultimately, you have to ask yourself, if someone robs, attacks, threatens, sometimes kills, others, out of nothing else but self benefit, or some petty shit, how does this person "want" to be part of society? Why should this person even have access to that option after behaving in such ways?

This is how you end like Europe, a corrupt bureocratic pigsty, infinite money for football stadiums, whore parties, dinners and lunches for the friends and the gang, doing the great business and a never ending expansion of the pig machine.... Age of retirement going up, houses out of range for most people, prices through the roof on everything, sale of profitable state enterprises (by these socialists, communists, leftist pigs), industries with years upon years of advantage getting taken for lunch by countries which got the technology from Europe and USA, no army, unable to spend 2% for self-defense... 2%... Again for football, pigsty non-profits, retards, cunts and dick suckers, shitty enterprises, for the friends, for COVID vaccinnes, for the professional commentators on TVs, journals, etc, there's always money for that. A woman from PSD was on the Parlament for 10 years got retired with a lifetime of 5.000€ monthly payments ... From the time she retired, taking into account life expectancy, she will get from the state in those remaining years more than 3 million €, paid by you, by doing nothing, after having been paid all those other years...

Isn't it interesting what these pigs did with Europe? Imagine a continent with this much potential, can't even fuck to replacement rates, more than half the population addicted (between alcohol, recreational and medical drugs) and the only ones interested in settling here are whores, pigs, criminals (from petty to high-crime) and people who come from places where they live worse than pets in the West... This is what Europe has become, the second biggest consumer market in the world can't even renew themselves... You know why? Just pigs and retards like you would like to live in the world you created, and this is what you see... It's people who have invested too much and can't get out (life, house, work, etc) the politicians and civil servants that have for life positions, those who can't get out and the drugged out retards that just go through life like some fucking automaton that can't even string a logical thought from A to C even though he spent 12 years on public education, 9-to-5. Really amazing.

Hope this expanded your view of the world and make you more aware of the reality in which you thread.

Today I'm going to write about a little Ecto type I use regularly in most Elixir projects I work in. Basically, it's an enum wrapper that allows you to use a limited set of :atoms or "strings" values in your Ecto schemas definitions.

You might be wondering why a writing an enum type when Ecto ships with an Ecto.Enum type since v3.5. The main reason is the typing of the "type" itself and sharing this type between multiple schemas.

Ecto.Enum is great and very handy for most regular cases where you simply want to be able to deal with atom versions of strings in your schemas and Elixir logic, while being able to store them as strings automatically at the database level and be able to cast from strings to atoms - which is a pretty normal usecase when dealing with external parameters or inputs.

To show you what I mean lets say you have a schema like:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, :string)
  end
end

Now, this is a totally valid schema. But if you have a limited set of values that :action might assume, right now you have no way of ensuring that when dealing with casting / inserting records of this type of schema the strings that are passed on are valid.

You might create this constraint at the database level, and if it's an essential part of your business logic you probably should keep alongside such constraint (although it makes it more onerous to update and change), but even if you don't, it would be nice to at least in your business logic be able to treat this :action field as having only a value from a set of known, well defined, values without having to explicitly read the field/param and manually checking it.

This is where Ecto.Enum can be used. Applying it to our schema means:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, Ecto.Enum, values: [:bid, :request, :upload, :pay])
  end
end

In this case, we basically defined as valid values only that specific set of atoms and so when casting against that schema field, any string value, or atom, that matches those atoms if converted would also be a valid value.

What this gives us is:

• the ability to cast params and make sure the action field is only valid if it conforms to those values, ex:

params = %{"user_id" => 1, "action" => "bid"}

Ecto.Changeset.cast(%User.Action{}, params, [:user_id, :action])

• The ability to deal strictly with the atom versions in our programs logic, by being sure that after casting or retrieving a record from the database it will have to conform, ex:

def log_action?(%User.Action{action: action}),
    do: action in [:bid, :pay]

And in many cases this is enough. But if we, for instance, want to use that same "action type" in another schema, then we have to duplicate its definition - and it's never really the same "type", we need to keep both (or more versions if used in more places) in synch. It also doesn't expose directly the normal cast/load/dump functions that can also be useful, in some contexts. For instance, you might want to cast a single string somewhere and depending on the success of such take a path on your code or another.

It also doesn't allow you to use integer versions for the underlying enum definition at the database level, nor do some useful tricks with casting & evaluating the value when dealing with "translations" - e.g. some external datasource uses some strings values that you don't want to store as they are, nor assume as valid for the field, but instead convert them to your app's version, or you simply assert their "equivalence", even if they have different representations - say one external source uses bidding but all others including yours use bid. You don't want to allow bidding to be stored or deal with it internally, but you want to be able to cast bidding as bid.

There's also the fact that it doesn't expose a @typespec that you can use, which can be useful as well, when using dialyzer (which you, most of the times, should be) or simply when writing @specs in your code's documentation (which you should always strive to do).

None of these are showstoppers though and if you don't need any of the additional functionality then just using Ecto.Enum is a significant improvement over bare strings and might cover all your requirements while being part of Ecto itself (meaning, no additional dependencies or code to maintain).

If we were to be writing a single Ecto.Type, following the examples before, we could do something such as:

defmodule User.Action.Type do
  @behaviour Ecto.Type

  def type, do: :string

  @valid_types [:bid, :request, :upload, :pay]
  @valid_strings Enum.reduce(@valid_types, [], fn t, acc -> [Atom.to_string(t) | acc] end)
  @valid_map Enum.reduce(@valid_types, %{}, fn t, acc -> Map.put(acc, Atom.to_string(t), t) end)

  @valid_inverse_map Enum.reduce(@valid_types, %{}, fn t, acc -> Map.put(acc, t, Atom.to_string(t)) end)

  @type t() :: unquote(Enum.reduce(Enum.reverse(@valid_types), &{:|, [], [&1, &2]}))

  def load(data), do: cast(data)

  def cast(data) when is_atom(data) and data in @valid_types, do: {:ok, data}
  def cast(data) when is_binary(data) and data in @valid_strings, do: {:ok, @valid_map[data]}
  def cast(_), do: :error

  def dump(data) when is_atom(data) and data in @valid_types, do: {:ok, @valid_inverse_map[data]}
  def dump(data) when is_binary(data) and data in @valid_strings, do: {:ok, data}
  def dump(_), do: :error

  def embed_as(_), do: :dump
  def equal?(data, data), do: true

  def equal?(data_1, data_2) do
    case {cast(data_1), cast(data_2)} do
      {{ '{{' }}:ok, same}, {:ok, same}} -> true
      _ -> false
    end
  end
end

Breaking it down:

• declare a behaviour of Ecto.Type for our module (this ensures we'll get warnings if we don't implement some functions that are expected to be defined for such behaviour)
• Declare a function type/0 that returns the underlying type of the Ecto.Type (what is used to persist values of this type in the database), in this case :string (which ecto uses to abstract over all char, varchar, and text column types)
• Declare a typespec, that returns the valid atoms this type represents
• From the list of possible atoms, we create a list of their counterparts in string form, and two module attributes to store the mappings between atom<->string versions and vice-versa
• Declare all required behaviour functions for the Ecto.Type, since we have lists of both atoms and strings representing values we consider valid, we can use these in guards, to cast them appropriately to their atom form when in string, and the same for dumping them from atoms to strings

As you can see it's easy to create an Ecto.Type that provides us with a more constrained set of values and helps us being more assertive on our code about what we expect.

It also makes it so that now we can actually cast those values independently of a schema. We can just do User.Action.Type.cast("bid") to validate if it's a valid action according to the type.

If you don't mind the copy pasting and changing the @valid_types list whenever you want to add a new type it's fine to use it like this. You can just use this as a blueprint then change the module name, and the list of values whenever you need to define another enum.

This already has some advantages, besides being able to cast the values independently of schemas, the type itself is now independent of their declaration in a schema, and as such can be used in multiple schemas as well.

We could also easily make it use integers for its underlying representation, as in a conventional enum. But we can also write a macro that effectively does all of this for us with a single line of code. Our objective is to have a macro that enables us to write the previous User.Action.Type module as:

defmodule User.Action.Type do
  use TypedEnum, values: [:bid, :request, :upload, :pay]
end

Followed by using it on the schema:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, User.Action.Type)
  end
end

And if we want to deal with atoms in our inner logic, but store them as integers in the database, while still allowing the casting of stringed versions:

defmodule User.Action.Type do
  use TypedEnum, values: [bid: 0, request: 1, upload: 2, pay: 3]
end

Let's breakdown what our module will need to do:

• Implement the __using__ macro, so that it can be "used" in the way we wrote in the previous examples
• Be able to distinguish between a plain list of atoms or a Keyword list being passed as values to allow both integer and string based enums at the database layer
• Allow the caller to write any additional cases for the cast\dump logic of the Ecto.Type behaviour
• Implement automatically a typespec for it, reflecting all the valid atoms it can assume
• Raise on invalid values being passed to provide compilation time blowups.

This is our final module and we'll break it down after:

defmodule TypedEnum do
  defmacro __before_compile__(_env) do
    # these are inserted in the before_compile hook to give opportunity to the
    # implementing module to define additional variations
    quote do
      def cast(_), do: :error
      def dump(_), do: :error
      defp get_term(data), do: data
    end
  end

  defmacro __using__(opts) do
    values = Keyword.fetch!(opts, :values)
    mod = __CALLER__.module

    is_int_enum? = Keyword.keyword?(values)

    case is_int_enum? do
      true ->
        :ok = validate_int_enum(values)
        bind_as_integer_version(values, mod)

      false ->
        :ok = validate_string_enum(values)
        bind_as_stringed_version(values, mod)
    end
  end

  defp validate_int_enum(values) do
    with {_, true} <- {:length, length(values) > 0},
         {_, true} <- {:format, Enum.all?(values, &valid_int_enum?/1)} do
      :ok
    else
        error -> raise_error(error)
    end
  end

  def validate_string_enum(values) do
    with {_, true} <- {:length, length(values) > 0},
         {_, true} <- {:format, Enum.all?(values, &is_atom/1)} do
      :ok
    else
        error -> raise_error(error)
    end
  end

  def valid_int_enum?({k, v}),
    do: is_atom(k) and is_integer(v)

  def raise_error({:length, _}),
    do: raise "TypedEnum expects `:values` to be a list or keyword list with at least 1 element"

  def raise_error({:format, _}),
    do: raise "TypedEnum expects the format of `:values` to be a keywordlist with the atom version as the key and an integer as the value (e.g.: [atom_key: 1, another_possible: 2, ...]), or a list of atoms for the string enum version (e.g.: [:atom_key, :another_possible, ...])"

  def bind_as_integer_version(values, mod) do
    quote bind_quoted: [atoms_ints: values, mod: mod] do
      @before_compile TypedEnum

      atom_integer_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, atom_val, int)
        end)

      string_integer_map =
        Enum.reduce(atom_integer_map, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, Atom.to_string(atom_val), int)
        end)

      string_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, _}, acc ->
          Map.put(acc, Atom.to_string(atom_val), atom_val)
        end)

      integer_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, int, atom_val)
        end)

      strings = Enum.map(atoms_ints, fn {atom_val, _} -> Atom.to_string(atom_val) end)
      atoms = Enum.map(atoms_ints, fn {atom_val, _} -> atom_val end)
      ints = Enum.map(atoms_ints, fn {_, int} -> int end)

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :integer

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :valid_ints, ints)
      Module.put_attribute(mod, :validation_mappings, string_atom_map)
      Module.put_attribute(mod, :validation_mappings_atoms, atom_integer_map)
      Module.put_attribute(mod, :validation_mappings_strings, string_integer_map)
      Module.put_attribute(mod, :validation_mappings_ints, integer_atom_map)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings | :ints) :: list(t()) | list(String.t()) | list(integer())
      @doc "Given a desired format returns the matching values for that format"
      def values(type \\ :atoms)
      def values(:ints), do: unquote(ints)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings[data]}

      def cast(data) when is_integer(data) and data in unquote(ints),
        do: {:ok, @validation_mappings_ints[data]}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, @validation_mappings_atoms[data]}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings_strings[data]}

      def dump(data) when is_integer(data) and data in unquote(ints), do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: @validation_mappings_atoms[data]

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings_strings[data]

      defp get_term(data) when is_integer(data) and data in unquote(ints),
        do: data
    end
  end

  def bind_as_stringed_version(values, mod) do
    quote bind_quoted: [atoms: values, mod: mod] do
      @before_compile TypedEnum

      strings = Enum.map(atoms, fn entry -> Atom.to_string(entry) end)
      mapped = Enum.zip(strings, atoms) |> Enum.into(%{})

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :string

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :validation_mappings, mapped)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings) :: list(t()) | list(String.t())
      def values(type \\ :atoms)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms), do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, String.to_atom(data)}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, Atom.to_string(data)}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: data

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings[data]
    end
  end
end

We declare the __before_compile__ macro that allows us to place some code in the module after everything else is in place, in this case we want to include some catch-all clauses. The reason we do it in the __before_compile__ hook is that this then enables a user of the module to implement their own versions of those functions, which can be useful for instance to include clauses that cast correctly values that come from third party systems that we can't change, but we do not want to allow on our db, instead translate them to our versions.

Because the load/1 function will already catch all possible terms and simply call cast/1 we do not include it there.

Then the actual __using__ macro.

  defmacro __using__(opts) do
    values = Keyword.fetch!(opts, :values)
    mod = __CALLER__.module

    is_int_enum? = Keyword.keyword?(values)

    case is_int_enum? do
      true ->
        :ok = validate_int_enum(values)
        bind_as_integer_version(values, mod)

      false ->
        :ok = validate_string_enum(values)
        bind_as_stringed_version(values, mod)
    end
  end

This is a macro, so it will be called at compile time. We fetch! the :values key from the options - since it's required we use the ! version in order to raise when it's missing. We then decide if the underlying enum will be integer or string based through a very simple heuristic. Are the values a keyword list or a plain list? If it's keyword list then it must be an integer based enum, otherwise a string based enum.

Depending on that we do a check on the form of the values to make sure they comply and then generate the actual code to fill the caller module. Remember that this is a macro, so these checks/raises will occur at compile time allowing us to signal to the user of our module any problem.

We'll only go through the integer version since it's the most complex, but you should be able to then understand easily the stringed version too. The validation helpers are plain functions, but since they're called in the context of a macro they will raise at compile time. Lastly, both bind_as_ functions do quote their contents so that these are placed in the calling module as code.

  def bind_as_integer_version(values, mod) do
    quote bind_quoted: [atoms_ints: values, mod: mod] do
      @before_compile TypedEnum

      atom_integer_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, atom_val, int)
        end)

      string_integer_map =
        Enum.reduce(atom_integer_map, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, Atom.to_string(atom_val), int)
        end)

      string_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, _}, acc ->
          Map.put(acc, Atom.to_string(atom_val), atom_val)
        end)

      integer_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, int, atom_val)
        end)

      strings = Enum.map(atoms_ints, fn {atom_val, _} -> Atom.to_string(atom_val) end)
      atoms = Enum.map(atoms_ints, fn {atom_val, _} -> atom_val end)
      ints = Enum.map(atoms_ints, fn {_, int} -> int end)

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :integer

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :valid_ints, ints)
      Module.put_attribute(mod, :validation_mappings, string_atom_map)
      Module.put_attribute(mod, :validation_mappings_atoms, atom_integer_map)
      Module.put_attribute(mod, :validation_mappings_strings, string_integer_map)
      Module.put_attribute(mod, :validation_mappings_ints, integer_atom_map)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings | :ints) :: list(t()) | list(String.t()) | list(integer())
      @doc "Given a desired format returns the matching values for that format"
      def values(type \\ :atoms)
      def values(:ints), do: unquote(ints)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings[data]}

      def cast(data) when is_integer(data) and data in unquote(ints),
        do: {:ok, @validation_mappings_ints[data]}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, @validation_mappings_atoms[data]}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings_strings[data]}

      def dump(data) when is_integer(data) and data in unquote(ints), do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
.            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: @validation_mappings_atoms[data]

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings_strings[data]

      defp get_term(data) when is_integer(data) and data in unquote(ints),
        do: data
    end
  end

We do bind_quoted passing the values as the variable atoms_ints and caller module as mod.

The generated code will include the @before_compile hook referencing this module where we defined it, TypedEnum. Then we use the keyword list of atoms_ints to generate a few different maps and lists that we'll use further down.

Because keyword lists of literal terms are literal proplists inside macros it makes them very ergonomic to use inside macros. They also provide a nice easy syntax to pass options as in this case.

So from the values keyword list (e.g.: [val_1: 1, val_2: 2]) we generate 4 maps:

• A simple map version of the proplist, the prop key as map key and the integer as the value, e.g.: %{val_1: 1, ...}
• Another in the same form but now the keys as strings, e.g: %{"val_1" => 1, ...}
• Another string to atoms mapping, e.g: %{"val_1" => :val_1, ...}
• And lastly a integer to atom mapping, e.g: %{1 => :val_1, ...}

These 4 maps will be used to fetch the right form of the enum depending on if we're loading or dumping it, and if the original data being passed in is in atom, string or integer form.

Then we generate 3 lists to aid us in adding guard clauses to our functions:

• for strings, e.g: ["val_1", ...]
• for atoms, e.g: [:val_1, ...]
• for ints, e.g: [1, ...]

We put all these things as module attributes (so the caller module can access them if they want, with the @... syntax) and we use them to provide the helper function values(type), that allows the user to retrieve the valid enum representations in any of its valid forms.

The cast functions of the Ecto.Type (and in our case the load) are used to transform values from their representation in the database, or from external requests, into our internal representation, which are atoms. Following this then means that if we trie to load or cast an atom that is a valid representation we just return that same atom since it's already in its correct form. If on the other hand it's a string (comes from an external request perhaps), or an integer (loaded from the database layer), we, with the aid of the mappings we assembled before, convert those into their atom version.

For the dump it's the same logic, but in this case the final form we want is the integer form, so we convert atoms and strings into their integer version, and if it's already an integer and a valid one we use that.

Otherwise, because of the guards we set up on each function, if the value they're called with is invalid, the catch all clauses that we set on the @before_hook will kick in and error at the ecto layer.

This allows still a user of the module to implement intermediate versions. Say one client that interacts with this is using a legacy version of the enums, where instead of "val_1" they were using "some_prop" as its identifier. We could set a specific clause on our caller module handling that specific case, for example:

defmodule ExampleCallerModule do
  use TypedEnum, values: [val_1: 1, val_2: 2]

  def cast("some_prop"), do: {:ok, :val_1}
  def dump("some_prop"), do: {:ok, 1}
  defp get_term("some_prop"), do: :val_1
end

And this would allow us to cast it, and even test its equality automatically by defining as well the get_term version for it.

I also like to add a specific dump! version that just calls dump underneath and raises in case dump returns anything else than an :ok tuple. The get_term are just helpers to simplify the equality checks.

And this folks, is all there is to it. The stringed version is very very similar, but simpler because it only has to deal with atoms and strings.

I opted to keep them as two separate functions because although there's some overlap and duplication going on, I think that having a whole bunch of conditionals on the macro for generating functions depending on if it's the integer or stringed version, and different module attributes, etc would just make it less straightforward to read and change if the need arises.

Now that you've seen how it works, if you don't want to write it yourself you can just include the package I've you can find it here in hexdocs (or through github in mnussbaumer/typed_enum and use it straightaway. Otherwise hope you could take away something useful.

]]> https://micaelnussbaumer.com/posts/criar-um-ecto-type-como-enum Thu, 06 Jan 2022 16:30:00 +07 https://micaelnussbaumer.com/posts/criar-um-ecto-type-como-enum em inglês

Hoje vou mostrar um enumerador implementado como um Ecto.Type, que uso em quase todos os projectos de Elixir em que contribuo. É um tipo semelhante ao Ecto.Enum que permite utilizar átomos na lógica interna da aplicação, validar/transformar valores equivalentes na sua forma de caracteres, e ao mesmo tempo limitar os possíveis valores que determinado campo/coluna pode assumir. Como não há tradução actual para o tipo de "string", no sentido em que é utilizado em programação, irei utilizar o termo listagrama, no sentido que uma "string" é uma lista de caracteres, e gramma em latim é símbolo/letra/desenho (hexagrama, etc), ou seja, uma lista de "grammas".

Apesar do Ecto incluir um Ecto.Enum desde a versão 3.5, este sofre de alguns problemas:

• é declarado em conjunto com o esquema no qual é utilizado, o que não permite que seja partilhado facilmente entre esquemas,
• não gera informações de tipo para usar em especificações,
• não permite lógica adicional no tratamento de parâmetros adicionais,
• não permite utilizar números inteiros para a representação do mesmo ao nível da base de dados (apenas grupos de caracteres)

Mesmo assim é por si só melhor do que usar simplesmente string como tipo de coluna nos casos em que sabemos que a mesma apenas pode assumir uma lista limitada de valores. Para ilustrar isto, vejamos o seguinte módulo:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, :string)
  end
end

Isto é um esquema totalmente válido, mas tanto quanto sabemos ao lê-lo é que o campo :action é um grupo de caracteres arbitrário. Neste caso, :action apenas pode assumir um número limitado de valores mas o esquema em si não nos transmite essa informação de forma alguma.

Claro que poderíamos criar restrições ao nível da base de dados, e em certos casos independemente de tudo o resto devemos certamente fazê-lo, mas isso continuaria a não providenciar informação alguma a nível do código da nossa aplicação sobre as possíveis restrições, nem ajudaria a transformar ou traduzir valores entre sistemas externos e a nossa representação interna para esses mesmos valores. Para obter isso necessitamos de emergir ao nível dos nossos esquemas quais são os valores possíveis para tais colunas.

Nesse caso, o Ecto.Enum ajuda-nos a fazer exactamente isso, por exemplo, utilizando o mesmo esquema anterior, mas agora atribuíndo um Ecto.Enum como tipo ao invés da simples :string

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, Ecto.Enum, values: [:bid, :request, :upload, :pay])
  end
end

Agora é visível que valores este campo pode assumir. Simplesmente por ler a definição do esquema sabemos que são :bid, :request, :upload e :pay.

Este tipo também nos permite então traduzir valores no formato de simples caracteres para o nosso formato interno de átomos, o que é bastante útil uma vez que se tivermos de tratar de validar submissões de dados externas ao nosso sistema, essas submissões serão certamente no formato de caracteres e não em tipos nativos a Elixir/Erlang, como é o caso de átomos. No entanto, após traduzidos, podemos ter a certeza que são não só válidos (há uma equivalência explicíta entre ambas as representações) mas que podemos escrever a nossa lógica interna com recurso apenas a átomos, o que para além de ser mais eficaz é também mais robusto.

Resumindo:

• dá-nos a habilidade de transformar e validar parâmetros:

params = %{"user_id" => 1, "action" => "bid"}

Ecto.Changeset.cast(%User.Action{}, params, [:user_id, :action])

• permite-nos, como mencionado anteriormente, usar estritamente a versão de átomo na nossa lógica interna, pois sabemos que após ser validado terá de conformar-se a essa representação:

def log_action?(%User.Action{action: action}),
    do: action in [:bid, :pay]

Como átomos em Elixir e Erlang são internamente representados como números inteiros, é bastante mais performante utilizá-los, tanto para serializar ou de-serializar. Grupos de caracteres necessitam de muito mais trabalho para serem comparados e avaliados.

O que o Ecto.Enum adiciona é por si só em grande parte dos casos suficiente. É sempre melhor do que usar definições de tipos que permitem grupos arbitrários quando sabemos que apenas determinados valores são válidos.

Mas se por exemplo, agora quisessemos utilizar este mesmo tipo que definimos no User.Action noutro esquema, teríamos de duplicar esta definição, e não haveria ligação efectiva entre ambos, a não ser que fossemos buscar os valores a algum módulo/função partilhada e, mesmo nesse caso, a ligação não é formal, o tipo não é o mesmo nem é possível representá-lo como tal.

Para além disso, não temos também acesso a funções de validação independentes do esquema. O tipo sendo declarado no esquema em si, é parte do mesmo e as suas validações ocorrem no contexto de um esquema de Ecto. Adicionar claúsulas específicas para tratar de casos especiais é também impossível ou bastante mais complicado do que necessita de ser.

Adicionalmente, caso queiramos guardar tais valores como números inteiros ao nível da base de dados, também não o podemos fazer uma vez que o Ecto.Enum assume apenas listagramas (strings) como a coluna ao qual está associado.

Por último, não gera um typespec para o tipo. Caso usemos dialyzer na nossa aplicação não temos forma de transpor as restrições definidas no esquema para especificações de tipos para documentação (e consistência de documentação das interfaces).

No entanto é, como referido, bastante melhor que utilizar apenas :string como tipo, e por vezes também não necessitamos de mais nenhuma funcionalidade.

Caso necessitemos, vejamos primeiro como poderíamos definir um Ecto.Type independente para este caso em particular :

defmodule User.Action.Type do
  @behaviour Ecto.Type

  def type, do: :string

  @valid_atoms [:bid, :request, :upload, :pay]
  @valid_strings Enum.reduce(@valid_atoms, [], fn t, acc -> [Atom.to_string(t) | acc] end)
  @valid_map Enum.reduce(@valid_atoms, %{}, fn t, acc -> Map.put(acc, Atom.to_string(t), t) end)

  @valid_inverse_map Enum.reduce(@valid_atoms, %{}, fn t, acc -> Map.put(acc, t, Atom.to_string(t)) end)

  @type t() :: unquote(Enum.reduce(Enum.reverse(@valid_atoms), &{:|, [], [&1, &2]}))

  def load(data), do: cast(data)

  def cast(data) when is_atom(data) and data in @valid_atoms, do: {:ok, data}
  def cast(data) when is_binary(data) and data in @valid_strings, do: {:ok, @valid_map[data]}
  def cast(_), do: :error

  def dump(data) when is_atom(data) and data in @valid_atoms, do: {:ok, @valid_inverse_map[data]}
  def dump(data) when is_binary(data) and data in @valid_strings, do: {:ok, data}
  def dump(_), do: :error

  def embed_as(_), do: :dump
  def equal?(data, data), do: true

  def equal?(data_1, data_2) do
    case {cast(data_1), cast(data_2)} do
      {{ '{{' }}:ok, same}, {:ok, same}} -> true
      _ -> false
    end
  end
end

Analisando o módulo:

• declaramos que o módulo implementa o comportamento (@behaviour) de um Ecto.Type (o que nos garante avisos e erros aquando da compilação no caso de nos esquecer-mos de implementar alguma função)
• declaramos uma função type/0 que resolve no tipo interno que o módulo representa (o que será utilizado aquando da serialização para a base de dados). Neste caso resolve em :string (ou seja, a coluna na base de dados será uma coluna de texto - varchar, char, text, etc)
• declaramos um typespec, que resolve numa lista contendo todos os átomos válidos para este tipo
• utilizando essa lista de átomos válidos (@valid_atoms) geramos uma lista de listagramas que correspondem directamente às respectivas versões dos átomos, assim como dois atributos do módulo para guardar o mapeamento entre os mesmos, átomos->listagramas e listagramas->átomos
• declaramos todas as funções requeridas pelo comportamento Ecto.Type - uma vez que temos uma lista compostas por todos os átomos e outra por todos os listagramas válidos, podemos utilizar essas listas como cláusulas de restrição nas funções requeridas e dessa forma facilmente definir como se devem comportar.

Como podemos ver, é simples criar um Ecto.Type que represente um grupo restrito de valores, ajudando-nos a tornar o nosso código mais assertivo e explícito (e por conseguinte mais compreensível também).

Permite-nos também transformar e validar termos independentemente de um qualquer esquema. Podemos por exemplo executar User.Action.Type.cast("bid") para validar o termo "bid", que neste caso seria válido visto ser a representação em listagrama do átomo :bid.

Caso copiar ou replicar este módulo com as devidas alterações sempre que necessitemos de um tipo novo para um outro grupo de valores não for um problema, podemos utilizar esta versão e fazer exactamente isso, utilizando isto como a "planta" para todos os outros.

Esta versão já demonstra algumas vantagens relativamente ao Ecto.Enum, nomeadamente a capacidade de fazer transformações & validações de termos independentemente de esquemas, assim como ser utilizado em vários esquemas, sem requerer duplicação de código ou código adicional para os manter actualizados - futuras alterações aos valores do tipo ocorrem apenas no módulo que o define.

Uma vez que controlamos também o funcionamento interno do mesmo poderíamos facilmente modificá-lo para que serializasse os valores como números inteiros. No entanto, o melhor seria criar uma macro que faça tudo isso por nós, permitindo-nos aceder essa funcionalidade com uma simples linha de código. Tendo isso em mente, o nosso objectivo é criar uma macro que nos permite escrever o módulo User.Action.Type anteriormente exposto, como:

defmodule User.Action.Type do
  use TypedEnum, values: [:bid, :request, :upload, :pay]
end

De forma a utilizá-lo num esquema de Ecto da seguinte forma:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, User.Action.Type)
  end
end

E caso queiramos lidar com átomos ao nível da aplicação mas serializá-los enquanto números inteiros, que possamos definir o mesmo tipo como:

defmodule User.Action.Type do
  use TypedEnum, values: [bid: 0, request: 1, upload: 2, pay: 3]
end

Vejamos então uma lista do que o nosso módulo necessita suportar:

• a macro __using__, de forma que os módulos que queiram fazer uso dele, possam utilizar a palavra-chave use, como demonstrado nos exemplos do que queremos
• que seja capaz de distinguir entre listas de átomos como valores válidos e listas de palavras-chaves, de forma a que possa intuitivamente implementar ora versões com base de número inteiro, ou de listagramas
• permitir a quem o deseje implementar a possibilidade de escrever funções adicionais para cast/dump do comportamento Ecto.Type
• introduzir automaticamente um typespec (especificação de tipo) que reflita todos os valores válidos na sua forma de átomo
• que emerja erros aquando da compilação quando os valores utilizados na definição do tipo sejam inválidos

Dito isso, o nosso módulo final terá a seguinte forma:

defmodule TypedEnum do
  defmacro __before_compile__(_env) do
    # these are inserted in the before_compile hook to give opportunity to the
    # implementing module to define additional variations
    quote do
      def cast(_), do: :error
      def dump(_), do: :error
      defp get_term(data), do: data
    end
  end

  defmacro __using__(opts) do
    values = Keyword.fetch!(opts, :values)
    mod = __CALLER__.module

    is_int_enum? = Keyword.keyword?(values)

    case is_int_enum? do
      true ->
        :ok = validate_int_enum(values)
        bind_as_integer_version(values, mod)

      false ->
        :ok = validate_string_enum(values)
        bind_as_stringed_version(values, mod)
    end
  end

  defp validate_int_enum(values) do
    with {_, true} <- {:length, length(values) > 0},
         {_, true} <- {:format, Enum.all?(values, &valid_int_enum?/1)} do
      :ok
    else
        error -> raise_error(error)
    end
  end

  def validate_string_enum(values) do
    with {_, true} <- {:length, length(values) > 0},
         {_, true} <- {:format, Enum.all?(values, &is_atom/1)} do
      :ok
    else
        error -> raise_error(error)
    end
  end

  def valid_int_enum?({k, v}),
    do: is_atom(k) and is_integer(v)

  def raise_error({:length, _}),
    do: raise "TypedEnum expects `:values` to be a list or keyword list with at least 1 element"

  def raise_error({:format, _}),
    do: raise "TypedEnum expects the format of `:values` to be a keywordlist with the atom version as the key and an integer as the value (e.g.: [atom_key: 1, another_possible: 2, ...]), or a list of atoms for the string enum version (e.g.: [:atom_key, :another_possible, ...])"

  def bind_as_integer_version(values, mod) do
    quote bind_quoted: [atoms_ints: values, mod: mod] do
      @before_compile TypedEnum

      atom_integer_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, atom_val, int)
        end)

      string_integer_map =
        Enum.reduce(atom_integer_map, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, Atom.to_string(atom_val), int)
        end)

      string_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, _}, acc ->
          Map.put(acc, Atom.to_string(atom_val), atom_val)
        end)

      integer_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, int, atom_val)
        end)

      strings = Enum.map(atoms_ints, fn {atom_val, _} -> Atom.to_string(atom_val) end)
      atoms = Enum.map(atoms_ints, fn {atom_val, _} -> atom_val end)
      ints = Enum.map(atoms_ints, fn {_, int} -> int end)

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :integer

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :valid_ints, ints)
      Module.put_attribute(mod, :validation_mappings, string_atom_map)
      Module.put_attribute(mod, :validation_mappings_atoms, atom_integer_map)
      Module.put_attribute(mod, :validation_mappings_strings, string_integer_map)
      Module.put_attribute(mod, :validation_mappings_ints, integer_atom_map)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings | :ints) :: list(t()) | list(String.t()) | list(integer())
      @doc "Given a desired format returns the matching values for that format"
      def values(type \\ :atoms)
      def values(:ints), do: unquote(ints)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings[data]}

      def cast(data) when is_integer(data) and data in unquote(ints),
        do: {:ok, @validation_mappings_ints[data]}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, @validation_mappings_atoms[data]}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings_strings[data]}

      def dump(data) when is_integer(data) and data in unquote(ints), do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: @validation_mappings_atoms[data]

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings_strings[data]

      defp get_term(data) when is_integer(data) and data in unquote(ints),
        do: data
    end
  end

  def bind_as_stringed_version(values, mod) do
    quote bind_quoted: [atoms: values, mod: mod] do
      @before_compile TypedEnum

      strings = Enum.map(atoms, fn entry -> Atom.to_string(entry) end)
      mapped = Enum.zip(strings, atoms) |> Enum.into(%{})

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :string

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :validation_mappings, mapped)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings) :: list(t()) | list(String.t())
      def values(type \\ :atoms)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms), do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, String.to_atom(data)}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, Atom.to_string(data)}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: data

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings[data]
    end
  end
end

Para que quem queira utilizar esta macro tenha a possibilidade de incluir também cláusulas específicas para cast/dump/get_term, definimos uma macro especial, __before_compile__ que nos permite inserir código após o módulo ter sido compilado, mas antes da compilação ser dada como finalizada. Neste caso, através disso incluímos versões das funções load/dump/get_term que funcionam para quaisquer valores não válidos e resolve no átomo :error (que é o esperado pelo Ecto quando um valor não é válido).

Uma vez que a função load/1 simplesmente executa a função cast/1 não há necessidade de definir uma versão de erro para ela.

A, também especial, macro __using__:

  defmacro __using__(opts) do
    values = Keyword.fetch!(opts, :values)
    mod = __CALLER__.module

    is_int_enum? = Keyword.keyword?(values)

    case is_int_enum? do
      true ->
        :ok = validate_int_enum(values)
        bind_as_integer_version(values, mod)

      false ->
        :ok = validate_string_enum(values)
        bind_as_stringed_version(values, mod)
    end
  end

Uma vez que é uma macro, irá ser avaliada aquando da compilação do código. Nela, utilizamos Keyword.fetch! para o termo :values das opções que forem passadas ao use, por exemplo use TypedEnum, values: [:bid, :request, :upload, :pay] significa que a variável opts será uma lista Keyword com uma chave :values, cujo valor será uma lista de átomos (:bid, :request, etc). Uma vez que a chave :values é essencial ao funcionamento da macro, caso não esteja presente nas opts queremos que surja um erro, e daí utilizarmos fetch!.

De seguida decidimos se devemos definir o enumerador com valores de números inteiros ou listagramas, e fazemo-lo avaliando a forma do valor da chave :values. Caso seja uma lista Keyword assumimos que seja números inteiros, pois será composta por pares de versão_átomo -> número_inteiro_correspondente. Caso seja uma simples lista de átomos, assumimos que o desejado é que o enumerador use listagramas como formato.

Dependendo disso fazemos algumas verificações para nos certificarmos que os valores têm o formato correcto. Tudo isto acontece aquando da compilação, logo qualquer erro que haja emergirá nessa altura, o que permite ao utilizador saber imediatamente se há algum problema.

Agora iremos fazer uma leitura do que acontece na versão com base em números inteiros. Visto ser mais complexa que a com base em listagramas acredito que compreendendo esta será fácil ler a outra. Sendo uma macro que utiliza quote, os conteúdos dos blocos de quote do end irão ser introduzidos nos módulos que utilizarem esta macro e é o que se passa nesses blocos que é o mais importante.

  def bind_as_integer_version(values, mod) do
    quote bind_quoted: [atoms_ints: values, mod: mod] do
      @before_compile TypedEnum

      atom_integer_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, atom_val, int)
        end)

      string_integer_map =
        Enum.reduce(atom_integer_map, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, Atom.to_string(atom_val), int)
        end)

      string_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, _}, acc ->
          Map.put(acc, Atom.to_string(atom_val), atom_val)
        end)

      integer_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, int, atom_val)
        end)

      strings = Enum.map(atoms_ints, fn {atom_val, _} -> Atom.to_string(atom_val) end)
      atoms = Enum.map(atoms_ints, fn {atom_val, _} -> atom_val end)
      ints = Enum.map(atoms_ints, fn {_, int} -> int end)

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :integer

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :valid_ints, ints)
      Module.put_attribute(mod, :validation_mappings, string_atom_map)
      Module.put_attribute(mod, :validation_mappings_atoms, atom_integer_map)
      Module.put_attribute(mod, :validation_mappings_strings, string_integer_map)
      Module.put_attribute(mod, :validation_mappings_ints, integer_atom_map)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings | :ints) :: list(t()) | list(String.t()) | list(integer())
      @doc "Given a desired format returns the matching values for that format"
      def values(type \\ :atoms)
      def values(:ints), do: unquote(ints)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings[data]}

      def cast(data) when is_integer(data) and data in unquote(ints),
        do: {:ok, @validation_mappings_ints[data]}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, @validation_mappings_atoms[data]}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings_strings[data]}

      def dump(data) when is_integer(data) and data in unquote(ints), do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
.            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: @validation_mappings_atoms[data]

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings_strings[data]

      defp get_term(data) when is_integer(data) and data in unquote(ints),
        do: data
    end
  end

Neste bloco de quote usamos o atributo bind_quoted, que nos permite passar variáveis propriamente expandidas para serem utilizadas no interior do bloco. Neste caso, atoms_ints com os valores contidos em values (o que havia sido passado como chave :values à macro), e também mod, contendo o nome do módulo que executou a macro.

Primeiro incluímos imediatamente o atributo @before_compile, que faz com que após a compilação, mas antes da sua finalização, a macro __before_compile__ do módulo em questão seja executada, nesta caso o __before_compile__/1 do módulo TypedEnum. Depois utilizamos a lista de Keywords (atoms_ints) para gerar um conjunto de mapas e listas que nos auxiliarão a decidir que validações e transformações são necessárias quando as funções de comportamento Ecto.Type sejam executadas.

Dessa lista de values, geramos 4 mapas - se utilizarmos o exemplo anterior onde a chave :values assumia o valor de uma lista com a seguinte forma [val_1: 1, val_2: 2]), o resultado de atom_integer_map, string_integer_map, string_atom_map, integer_atom_map será respectivamente:

• mapa com as mesmas chaves e valores que a lista Keyword, neste caso: %{val_1: 1, ...}
• outro mapa idêntico mas agora com as chaves em formato de listagrama ao invés de átomos: %{"val_1" => 1, ...}
• um mapa que mapeia as correspondências entre listagramas e versões em átomos: %{"val_1" => :val_1, ...}
• por último, as correspondências entre números inteiros e as suas equivalências em formato de átomo, e.g: %{1 => :val_1, ...}

Com estes quatro mapas auxiliares será mais fácil resolvermos os valores do tipo dependendo se estamos a fazer uma validação/transformação (cast/1) ou uma serialização (dump/1), e tendo em conta o formato do termo original (n. inteiro, listagrama ou átomo).

De seguida geramos 3 listas adicionais, que nos facilitarão restringir as cláusulas das nossas funções:

• para listagramas: ["val_1", ...]
• para átomos: [:val_1, ...]
• para números inteiros: [1, ...]

Colocamos todas estas variáveis auxiliares em atributos do módulo (o que significa que quem esteja a utilizar o mesmo, os pode aceder com a síntaxe usual, @nome_do_atributo), e utilizamos a lista de valores válidos nos seus diferentes formatos para providenciar uma função, values/1, que resolve em todos os valores válidos para o formato desejado.

As funções cast (e no nosso caso load também), são usadas para transformar termos do seu formato interno na base de dados, ou externos à nossa aplicação, no formato utilizado pela nossa aplicação, neste caso em átomos. Por conseguinte, se parte do código por algum motivo executar load/cast com valores já no formato correcto, a resolução dessas funções resultará no mesmo termo. Se no entanto utilizarmos termos no formato de listagramas ou números inteiros, se tiverem uma correspondência válida, serão convertidos no seu formato de átomo. Caso sejam inválidos emergirão um erro.

No caso da função dump, a lógica é em tudo semelhante, com a diferença que o formato final que procuramos é o de número inteiro (para serialização na base de dados), neste caso, listagramas e átomos, se tiverem uma correspondência válida, serão convertidos no seu formato de número inteiro, caso o termo a avaliar seja já um número inteiro e válido, a função resolverá com esse mesmo termo.

Uma vez que cada função load/1 e dump/1 tem a acompanhar um grupo de restrições, caso os valores passados não sejam válidos, as cláusulas que inserimos através do @before_hook serão as executadas (uma vez que aceitam qualquer termo e não têm guardas), e resultarão num erro.

Caso o utilizador do módulo, tenha adicionado cláusulas extra, devido à ordem final de todas elas no módulo compilado, essas serão tentativamente executadas antes das dos erros. Por exemplo, imaginemos que estamos a programar uma aplicação que necessita interagir com uma versão antiga da base de dados, em que ao invés de utilizarem o termo "val_1", estavam a utilizar o termo "valor_1". Actualmente "valor_1" já não é uma representação válida, por isso não queremos permitir que seja persistida pelo nosso sistema, mas ao mesmo tempo, tendo uma correspondência directa ao termo "val_1" que por sua vez é válido, queremos convertê-la caso ela surja como parâmetro em alguma situação.

Nesse caso poderiamos definir o nosso tipo da seguinte forma:

defmodule ExampleCallerModule do
  use TypedEnum, values: [val_1: 1, val_2: 2]

  def cast("valor_1"), do: {:ok, :val_1}
  def dump("valor_1"), do: {:ok, 1}
  defp get_term("valor_1"), do: :val_1
end

E isto permitir-nos-ia validar, traduzir e verificar a sua equivalência, sem no entanto necessitar de introduzir mais um item no enumerador, ou complicar significativamente o nosso programa de forma a suportar tal caso.

Adicionalmente às funções requeridas pelo comportamento Ecto.Type eu pessoalmente gosto de definir também uma dump!/1, que simplesmente executa a dump/1 normal, e caso a resolução seja qualquer outra coisa que uma tupla {:ok, _}, que emerja um erro.

A função get_term é simplesmente para facilitar a verificação de equivalência entre termos.

E isto é basicamente tudo o que é necessario. A versão em listagramas é em todo símile a esta, mas mais simples uma vez que apenas necessita de funcionar com átomos e listagramas.

Como última nota relativa ao módulo que escrevemos, optei por manter duas versões separadas para a versão listagrama e outra para a versão de números inteiros - isto porque apesar de haver partes duplicadas entre ambas, escrever apenas uma versão requereria uso de várias condicionais, e penso que o código resultante seria bem mais complicado e não linear de entender. Assim temos um pouco de duplicação mas cada parte trata apenas do que tem a tratar.

E agora que vemos como poderíamos implementar tal módulo para nos auxiliar, caso queiram podem utilizar um pacote já feito, disponível em hexdocs (ou github mnussbaumer/typed_enum. De qualquer das formas espero que tenha sido útil.

]]> https://micaelnussbaumer.com/posts/what-s-wrong-with-software-development Tue, 16 Nov 2021 01:00:00 +07 https://micaelnussbaumer.com/posts/what-s-wrong-with-software-development (Addendum 2026-03-14: This was written before the current AI cycle but funnily, it seems, most of these things turned out, or are turning out, to be also important for Agentic coding, which is kinda poetic...)

---

As someone who writes code for most of my time, either as my work or in personal projects I find it extremely difficult to understand the lacking of proper written documents describing what should be accomplished and the reluctance in using the written format to discuss and document the development of software.

First let's go through what I called "written documents". There are different forms these can assume, in common they have that they specify to varying degrees of detail, technicality, and/or context the purpose, specifications, and requirements of a given piece of "software idea".

I call it "software idea", because that's what most things are until they're fully materialized as "working software".

Some of the more common documents (or artefacts) that concern this are:

• - Written Specifications:
• Document(s) detailing, usually with high attention to technical capabilities, restrictions or requirements, in an unambiguous language, the working of an intended software program
• - User Stories:
• Written documents detailing at a high level the intended functionality of a software program/capability, from the point of view of a user of the software interfaces
• - Design Documents:
• - Visual document, either in the form of diagrams, and/or actual illustrations of the intended software

There's probably more ways, or specific terms to specify what a system, functionality, or capability is to achieve, both in itself or when in bigger existing system (when it's not self-contained) but these cover most of the "ideas".

I don't expect everybody (including myself) to master all these forms of laying out projects, in all contexts and all situations. If I'm working with someone who is not a software developer and just wants a website that does X, I know for sure I will have to weight in my own knowledge, and do extra work to get into the details of what is wanted and access its feasibility. It's normal in these cases to have to go from simple user stories, and it's expected that you as the developer will take care of the technicalities related to implementing them.

If I'm working through a Project Manager (PM) then my expectations change a little bit. I'm no longer working with someone that simply wants an idea converted into software, but instead someone that, as the title implies, is responsible for managing a project, and as such I expect them to have both technical knowledge and to dedicate part of their time producing more relevant and accurate documents detailing what is to be implemented. Again, it's not reasonable to expect that a PM will know in all details both the underlying technology and any and all roadblocks, but it's normal to expect a well rounded specification and at least some understanding of the system, languages and platforms in use. This last part is relevant and I'll touch on it later on.

---

We'll focus more on the software being developed with a PM, since the other cases are in themselves a bit off-target when discussing the need for documentation and written artefacts besides the code. Most projects, even small ones, will usually have a PM that is in charge of overseeing the implementation of whatever is to be built.

Let me start by prefacing that I understand that a PM has a non trivial amount of things to worry about - managing bikeshedding, going through discussions full of cruft, making sure development is aligned with upper goals, making sure it's on track, solving issues, allocating resources and making technical decisions. Let me also add, that although I don't claim that I would do their work better than they do, their work is their work and not mine. If I was to be hired or paid to both oversee and develop (i.e. code), then my compensation needs to take that into account and in case it does, I will be more than happy to take on additional responsibilities. If on the other hand my compensation doesn't have equity and is priced at a time unit, then I don't think it's reasonable to expect me to take on additionaly responsibility that someone else is being paid to be responsible for. As with software, good separation of concerns is important. I will do my best nonetheless, but if your advice, suggestions and observations are continuously ignored irrespective of their value (either immediate or solving a problem that manifests itself later on) at some point you just don't do that anymore.

The same if after careful consideration, and even spending "non-billable" time on a problem, your work is rejected and replaced with some dim-wit solution that "is simpler", and discards all the knowledge you had gathered while solving it (here I'm not saying in the sense of being attached to the code you write, there's certainly cases where you come up with a solution that is more complex than it needs to be and it's welcomed in that case that a PM or team mate rejects it, it's part of their work and part of being in a team, I'm refering to ditching something for something else that has problems from the get-go, not only imaginary future problems).

So it's in this situation, working with a PM, that the reluctance in requiring and expecting written artefacts to be produced becomes interesting. In a way I would expect, and if I work as a PM would require, that documentation (discussions, specifications, guides, how-to's related to the project, etc) AND tests are something fundamental. As a PM I can't think of any other way of keeping up with how everything works, how everything is SUPPOSED to work, and how we got into this given place.

People insist on doing video/call based meetings where the "team" catches up and courses of action are decided. This is ok to do, and there might be a need to even have these outside of pure business/development concerns (as team building rituals not concerned with specific work related issues) but it can't replace written documentation.

Folks whine about tacit knowledge but then don't take any action to solve it which, guess what, can only be solved with written documentation. People will prefer re-writing a completely correctly implemented solutions, with all the costs that entails, because there's no one that knows how it works. PM's end up having no frigging clue of how the project they're managing works, is tied (or glued) together, all because they can't be bothered to go through more than 1 paragraph of text.

---

Here I will add, that developers are some times actively part of the problem as well. The PM's work is nonetheless making sure this is taken care of, not making it normal though. We can put in the list of developer-made problems, the unwillingness to learn specifics outside their area of comfort. Examples can be something that is written in a language that they don't have as much familiarity with. You sometimes see perfectly valid, tested, extensible code, be replaced by orders of magnitude more complexity, untested, brittle layers, just for the sake of sidestepping having to learn how something is being accomplished with much less. No documentation, no testing, no how-to's for getting pieces of a system working, etc. Although these are all problems a developer contributes too, the role of the PM should be to make sure these things are required and properly budgeted. At least for me, it's fairly easier to document something as I'm writing it as opposed to after the fact.

This lack of written documentation (or performative documentation, such as tests - in the sense that it performs work to assert the validity of its contents) then exacerbates the issues with keeping up with how things work, if they should be working, and having a complete view of a system. On top of that, it makes on-boarding fairly more difficult as one can't for the most part just go on "discoverability" mode and instead has to rely on lower level code to gather what the "systems" should be doing. It's also fairly more difficult to access if a system is complex because of the domain it covers or if because it's being wrongly thought about.

Things such as refactoring become a painful chore, because, why would you try to refactor something when you don't even have a baseline of expected functionality asserted? Or at least something explaining what that functionality is expected to do? So if it's not product breaking it just gets pushed to the "backlog" void. If it's product breaking then all the required (like, big picture at least) tests to make sure the refactoring conforms have to be written - probably by someone who didn't write the initial implementation, doesn't have all the context, nor knows many of the pitfalls encountered during development because, you guessed it, there's no documentation.

Then, when the backlog void threatens to suck up the whole thing, the solution is to re-write. On top of this you might also have developers just deciding to quit or unable to have any semblance of productivity.

So when people go about how they don't have budget for documentation, testing, to put effort into making your software understandable what they're actually saying is they don't care at all for the costs of developing software. Sometimes this makes sense, again, if it's a online freelancing gig for a test-the-waters website, sure, go ahead - it might not even work so those costs would all be extra. But even in those situations, if the software is expected to have some modicum of complexity, tests and documentation will inevitably pay themselves in anything running for more than 2 months, or when you have to have another developer picking up from the existing work.

---

What is really mind-boggling is that having documentation and written word as first-class artefacts in a project as the baseline helps beyond these directly related issues already mentioned.

Jobs will have descriptions such as "employee budget for improvement/knowledge material/courses", "attending conferences", etc. But just writing proper documents that are to be used by people of varying technical literacy is a valuable skill. No-one is expected to do all things required for software development super-well, but this is probably one of those things that translates into any field of work, any language stack, any problem domain.

(Addendum 2026-03-14: This is one of the things that seem to be very important for Agentic coding...)

Then you'll see companies hiring for remote positions, but they can't produce any written documents, or specifications. In terms of hiring I also think that this is something that can be greatly improved by having a culture of written first. If your processes, your software, your systems, are documented, it's fairly more easy to:

• Produce a specification for interview takeaway problems (either new, or using something already implemented, in that case if you have tests then you can automatically measure the functionality of the returned solutions and then look into subjective aspects such as code organisation and others) - this is enabled because it's normal to specify things, and so it's easier to generate a new one because PMs, software engineers `(?!)`, and others have at least some experience doing it;
• Communicate effectively asynchronously - if your company can't produce written documentation, can't communicate clearly through written form, how am I supposed to believe somehow you are able to foster a remote environment that has any semblance of working?
• Onboard new people - yes, nothing replaces being able to talk directly to a person and have a question answered on the spot - the issue is that given the complexity of software systems, there is a lot of groundwork that needs to be done when coming up to speed - if the answers for 70% of it are written and discoverable, it's better than 0% and 100% of the questions needing some person to answer them, or having to figure them out by backtracking through actual code, instead of clearer high-level descriptions;
• Inter/intra team communication - if nothing is written, again the overhead of sharing knowledge is extremely high - some people will also be reluctant to engage if the other person doesn't show that at least they tried understanding something first - written documents can help this as well, since someone can do their own research and be left only with questions related to details - or not have questions afterall - it also shows how the documentation might be improved or is somehow missing so effective documentation improvements can be fed into these loops;
• Lastly, and this is perhaps personal, I think it's also a proxy for team fit - not a fit into a particular team, but fit into the idea of being in a team - people who take care in explaining how things work in order to be grokkable by others seem to me more inclined to care less about politics and not of being in a position just to take a payment at the end of the month/sprint - basically because by making their work accessible to others they're removing the need to rely on their tacit knowledge.

---

This is why to me it's a bit mind-boggling that it's not a core part of software development team processes. It's not only for developers, it's for PMs too and other members. It's not only about facilitating knowledge sharing and reducing bus factor, it's also about improving skills that are useful in a wide range of fields beyond software development (the ability to express ideas in the written form - because that scales). It's not only about empowering developers but about steering product development to a model where things are based more on their merits than on who has the most clout, political or personal, or who is the smartest-talking ape. It's also funny that people eagerly will copy tech-stacks and easy to do rituals (like stand-ups, daily calls, etc) from successful companies, but then the things that are far-reaching, such as emphasis on written documentation/preparation/spec'ing, are just ignored, probably because you can't just tick an item off a list and have to actually produce usable artefacts.

So the take-away I want to leave you with is; if you want to improve the quality of software that you or your team is producing, you want to empower and make a more resilient team, and diminish the long-term cost of software - start by documenting, spec'ing, testing and making this part of the normal expected workflow.

Most software might start trivial, or start as different services trivilly glued together, but complexity grows exponentially as it develops - not only because it has to take care of much more use cases as it expands, but because it usually has to rely on an increasing number of moving parts, from infrastructure, to packages, libraries, different languages, deployment targets, requirements and constraints and the costs of lacking "written artefacts" remain hidden. A developer that quits because everything is an ungrokkable mess doesn't show up as a direct cost of this. A developer having their productivity hampered in half doesn't show up as this. The need for a re-write because no one understands the system anymore doesn't show up as a cost of this. A new-hire having to spend valuable time from other more knowledgeable developers, that could easily be documented, doesn't show as related to this. Your PMs making wrong decisions doesn't either.

In the end, most of this is related to two things:

• the lack of willingness to admit that the written part of software development is essential - which shouldn't be a suprise since what we do as developers is writing code, sometimes comments, and we all know that the difficult part is clarifying the logic of what something should do and not the syntax specifics of given languages;
• the build to be acquired mentality - it doesn't matter if it's a turd, as long as you can make it look nice on numbers - many endeavours in software seem to only have the goal of being sold - and when that is the case, effectively, these might be additional costs and just feature after feature in a checklist, even if the base is a steeming pile, make it look more palatable - the truth is, it can also sink these ones, there's a breaking point after which you might not be able to have functioning systems and produce new features, and no matter how you sell it, it will show.

---

(of course, as with everything, there needs to be a balance, the same way you don't need to test everything that is in your code base to have a good degree of confidence, you might only test the overall code "paths" in some end-to-end fashion, and then specific functionality isolatedly, some things are self documenting, but I would arguee that many, surprisingly, are not, for anyone else than the one writing them - and that has a time validity as well, just like yogurt, after which they too, no longer understand their "clear self-documenting code")

]]> https://micaelnussbaumer.com/posts/imageengine-w-nextjs Thu, 10 Jun 2021 16:30:00 +07 https://micaelnussbaumer.com/posts/imageengine-w-nextjs Using ImageEngine in a React.js project

In this blog post we'll cover how to use ImageEngine in order to significantly improve your image assets, increasing your site responsiveness and improving your loading times - easy mode.

You can check the final website we'll be building (requires JS, no cookies)

There's also a repository containing all the code.

This post was originally published on dev.to

Why use ImageEngine (IE) to optimize your images?

While this post is particular to Next.js when exporting a static website - a case where Next.js inbuilt Image Loader, and its optimisations can't be used - IE only requires a query string on your image urls and can be used in any stack, even outside websites.

Next.js also has the capability of specifying custom image loaders for their inbuilt <Image/> component, and we can use those to reach an external CDN from which to load our images with our own logic - we'll go through that in another blog post.

If using Next.js with a Nodejs server and their inbuilt image optimisations, both the image optimisations and the serving of the assets will be done from the server side, while using IE none of it has to pass through your server with exception of the first serving - that coupled with the full range of customisable properties and highly optimised algorithms IE offers, makes it worth considering.

The only exception is in case you're deploying directly in vercel.com, their whole infra-structure supports Next.js inbuilt Image Optimizations (with automatic edge caching and good performance overall) which is good enough for most cases. Although even then, IE can provide significant bandwidth and improved response times savings over it, specially if you're serving a non trivial amount of images.

I'm going to walk you through implementing the website and deploying it on S3, using IE to optimise the delivery of its image content.

How can we use IE to improve our Nextjs app loading speed?

First let's go through the website so that you can make sense of what's happening and what we'll be building.

The website is designed to show the difference in file sizes, and consequently in loading times, between images distributed in their original versions and images distributed through ImageEngine's smart CDN. Notice that the images that comprise the repository are sourced from the same files, but by using IE we can do some neat things with them.



The thumbnails on the left are from images on the IE CDN, optimised through simple URL directives. This means that specifications such as width, height, compression, etc, are simply passed through query parameters on the URL the image tags use, without requiring any additional work.

The thumbnails on the right side are from the original resources.

If you click any thumbnail or file title from the sidebar, you'll see that for the IE side, the total size now increased to account for that, while the normal assets didn't - because they were already loaded.

Of note is that many times, when loading a website, there's no actual need of loading full sized images, so optimising thumbnails is immediately an advantage in the majority of cases - because it allows a much faster initial load it impacts directly Google’s Core Web Vitals scoring.

In the same way the thumbnails were optimised through directives, we can also optimise those being displayed on the "preview" area. From the sidebar select, Pick a type -> jpg, Pick a fit type -> cropbox and enter 600 as the width and 400 as the height directive. There's others, like compression, sharpness, and so on that aren't on the sidebar, but available to use.

You should see some significant improvements on the size without any perceivable quality loss. At 600px X 400px, it’s enough to cover retina when displaying images at 300px X 200px, which is roughly the size of our previews.

Click around on the thumbnails to see the time images take to be displayed between changes. Even though the regular images are already downloaded and cached by the browser you can still see significant differences (IE images on first load, or when changing directives have to be fully processed and transferred, and usually still render faster than the unoptimized ones already downloaded and cached!).

Let's start actually coding this.

Setting up the Next.js project

To follow through you'll need to have nodejs and npm installed. The versions used for this tutorial are npm 6.14.4 and node v10.19.0.

On the folder where you'll be creating your project run:

npx create-next-app

Now let's install the image engine package:

npm i @imageengine/react

We should be able to start our app by doing

npm run dev

And visiting

http://localhost:3000

should show us the default page. Let's remove the default templating index.js page has:

import Head from "next/head";

export default function Home() {
  return (
      <div className="main-container">
        <Head>
          <title>ImageEngine Optimized Assets</title>
        </Head>
      </div>
  );
};

And change /pages/_app.js to the following

import Head from "next/head";
import { ImageEngineProvider } from "@imageengine/react";
import "../styles/globals.css";

const delivery_address = process.env.DELIVERY_ADDRESS;

function LeanImages({ Component, pageProps }) {
    return (
	<ImageEngineProvider deliveryAddress={delivery_address}>
	  <Head>
	    <link rel="icon" type="image/png" href="/favicon.png"/>
	    <meta name="viewport" content="initial-scale=1.0, width=device-width" />
	    <meta name="description" content="Learn how to use ImageEngine in Nextjs to serve highly optimised image assets from your CDN to your users." />
	    <meta property="og:title" content="ImageEngine with NextJS" />
	    <meta property="og:description" content="Learn how to use ImageEngine in Nextjs to serve highly optimised image assets from your CDN to your users." />
	  </Head>
	  <Component {...pageProps} />
	</ImageEngineProvider>
    )
};

export default LeanImages;

If you have your dev server running you should see an error. Let's stop the server and create a next.config.js on the root of our project and inside it:

module.exports = {
    env: {
	DELIVERY_ADDRESS: process.env.DELIVERY_ADDRESS || "http://localhost:3000",
	NON_OPTIMIZED_ADDRESS: process.env.NON_OPTIMIZED_ADDRESS || ""
    },
};

Let's remove the things we don't need, delete pages/api folder, styles/Home.module.css and public/vercel.svg::

rm -rf pages/api rm styles/Home.module.css rm public/vercel.svg

We should now have the following structure:

pages /
  _app.js
  index.js
styles /
  global.css
public /
  favicon.ico

We're ready to start. Our website will be a single page, divided into 2 sections, one left sidebar and a main area, the latter further divided into 3 parts - the title and image information, the preview area, and the thumbnails section. From the sidebar we'll be able to choose certain parameters related to the image display, and we'll have a small list with actionable items.

When clicking one of those items it will display them in the main area, in two versions, one from our image engine distribution, and the other from our original assets.

Locally it will source both from the same place in our assets, but once online correctly from their origins.

We will also display the total transferred file size for each image and a cumulative for all images loaded, between optimized and non-optimized.

To share state between our components we’ll use useReducer so let's write its initial state, on our pages/index.js:

const initial_settings_state = {
    base_path: "/images",
    file_types: ["jpg", "jp2", "webp"],
    file_type: null,
    fit_types: ["stretch", "box", "letterbox", "cropbox"],
    fit_type: null,
    width: null,
    height: null,
    images: [
    	"senja_norway.jpg",
    	"buddha_shakyamuni.jpg",
    	"vasnetsov_samolet.jpg",
    	"great_wave_off_kanagawa.jpg"
    ],
    copyright: {
    	"great_wave_off_kanagawa.jpg": ["Great Wave off Kanagawa", "By After Katsushika Hokusai - Restored version, Public Domain", ["https://commons.wikimedia.org/w/index.php?curid=5576388"]],
    	"buddha_shakyamuni.jpg": ["Buddha Shakyamuni", "By Unknown author - Public Domain", ["https://www.metmuseum.org/collection/the-collection-online/search/75274", "https://commons.wikimedia.org/w/index.php?curid=39112914"]],
    	"vasnetsov_samolet.jpg": ["The Flying Carpet", "By Viktor Mikhailovich Vasnetsov - belygorod.ru, Public Domain", ["https://commons.wikimedia.org/w/index.php?curid=1374733"]],
    	"senja_norway.jpg": ["Island of Senja, Troms, Norway (August, 2014)", "By Ximonic (Simo Räsänen) - Own work, CC BY-SA 4.0", ["https://commons.wikimedia.org/w/index.php?curid=34693021"]]
	
    },
    show_copyright: false,
    loaded_images_ie: {},
    loaded_images_regular: {},
    selected_image: null,
    selected_image_url: null,
    force_load: false
};

Our reducer function will be:

function settings_reducer(state, action) {
    switch (action.type) {
    case "file_type":
    case "fit_type":
    case "width":
    case "height":
    case "force_load":
    case "show_copyright":
    	state[action.type] = action.value;
    	return {...state};
	
    case "selected_image":
    	state.selected_image = action.value;
    	state.selected_image_url = action.value ? `${state.base_path}/${action.value}` : null;
    	return {...state};

    case "add_image_size":
    	let state_key = build_type_key(action.image_type);
	
    	state[state_key][action.image] = action.size;
    	state[state_key] = {...state[state_key]};

    	return {...state};
    default:
        throw new Error();
  }
};

You might see an error because build_type_key(action.image_type) isn’t defined yet, we’ll add it in a while.

Sidebar component

Now our sidebar component. Create a folder components on the root folder and inside it a sidebar.js file:

import React from "react";

export default function Sidebar({ state, dispatch }) {
    return (
	<div className="sidebar-wrapper">
	  <div className="sidebar">
	    <h2>Directives Optimizations</h2>
	    <select id="file_type" name="file_type" value={state.file_type || ""} onChange={(event) => dispatch({type: "file_type", value: event.target.value})}>
	      <option value="" >Pick a type</option>
	      {state.file_types.map((type, index) => {
                return  <option value={type} key={`file_type-${index}`}>{type}</option>
	      })}
	    </select>
	    <br/>
	    <br/>
	    <select id="fit_type" name="fit_type" value={state.fit_type || ""} onChange={(event) => dispatch({type: "fit_type", value: event.target.value})}>
	      <option value="">Pick a fit type</option>
	      {state.fit_types.map((type, index) => {
                return  <option value={type} key={`file_type-${index}`}>{type}</option>
	      })}
	    </select>
	    <br/>
	    <br/>
	    <input type="text" name="width" id="width" placeholder="Width directive in px, e.g: 25" value={state.width || ""} onChange={(event) => dispatch({type: "width", value: event.target.value})}/>
	    <br/>
	    <br/>
	    <input type="text" name="height" id="height" placeholder="Height directive in px, e.g: 50" value={state.height || ""} onChange={(event) => dispatch({type: "height", value: event.target.value})}/>
            <ul>
              {state.images.map((image, index) => {
                return <li key={`image-${index}`} className={`${image === state.selected_image ? "selected" : ""}`} onClick={() => dispatch({type: "selected_image", value: image})}>{image}</li>
               })}
            </ul>
	  </div>
	</div>
    );
};

We haveselects, inputs and a list of elements. We set handlers to trigger on change for the inputs and dispatch the changes to our reducer, and click handlers on the list elements to dispatch the selection of an image.

Regarding the styling, to save some space I’ll link directly to the final css file. Either copy the whole file or otherwise add it piece by piece as I’ll mention the initial line (IL) and last line (LL) for each component. In this case IL0 - LL128.

The central part will be made of two sections, the top part - info and images previews - and the bottom thumbnail section. Create three files on the components folder, copyright.js, thumbnails.js and image_preview.js.

Copyright component

On copyright.js:

export default function Copyright({ state, dispatch }) {
    if (!state.selected_image) {
	return <div className="main-area-image-info"></div>;
    } else {
    	let copyright = state.copyright[state.selected_image],
    	    title = copyright[0],
    	    description = copyright[1],
    	    links = copyright[2];

    	return (
    	    <div className="main-area-image-info">
    	      <h4>{title}<button className="collapse-btn" type="button" onClick={() => dispatch({type: "show_copyright", value: !state.show_copyright})}>{state.show_copyright ? "-" : "+"}</button></h4>
    	      {state.show_copyright ? <div>
	             	    <p>{description}</p>
     			{links.map((link, index) => <a href={link} target="_blank" key={`link-${state.selected_image}-${index}`}>{link}<br/></a>)}
    	       </div> :
    	       null}
    	    </div>
    	)
    }
};

styles/globals.css add from IL131 - LL166.

Thumbnails component

thumbnails.js

import { Image } from "@imageengine/react";
import { get_size, handle_image, NON_OPTIMIZED_ADDRESS } from "../js/utilities";

export default function Thumbnails({ state, dispatch }) {
    
    return (
	<div className="main-area-thumbnails">
	  <div className="thumbnails-container optimized">
	    {state.force_load && state.images.map((image, index) => {
                 let image_path = `${state.base_path}/${image}`,
                     thumb_id = `thumb_${image_path}`;
                 return (
                    <div className="thumbnail-image" key={`thumbnail-optimized-${index}`} onClick={() => dispatch({type: "selected_image", value: image})}>
                      <div className="thumbnail-size">
                        {get_size(state, "ie", thumb_id)}
                      </div>
                      <Image
                        src={image_path}
                        directives={{
                          outputFormat: "jpg",
                          height: 90,
                          width: 90,
                          fitMethod: "cropbox",
                          compression: 80
                        }}
                        onLoad={(event) => handle_image("ie", event.target, thumb_id, dispatch)}
                      />
		    </div>
		);
	    })}
	  </div>
	  <div className="thumbnails-container regular">
            {state.force_load && state.images.map((image, index) => {
              let image_path = `${NON_OPTIMIZED_ADDRESS}${state.base_path}/${image}`;
              return (
                  <div className="thumbnail-image" key={`thumbnail-regular-${index}`} onClick={() => dispatch({type: "selected_image", value: image})}>
                     <div className="thumbnail-size">
                        {get_size(state, "regular", image_path)}
                     </div>
                     <img src={image_path} onLoad={(event) => handle_image("regular", event.target, image_path, dispatch)} />
                  </div>
                );
	    })}
	  </div>
	</div>
    )
};

Here we'll use the Image component provided by the @imageengine/react package, and import a bunch of helper functions not yet defined. We have two containers, on each of these containers we map the images we have to image elements - on the optimized one we build the full path, and create a unique id for this thumbnail. We display the size of that image with get_size and use IE’s Image component to build a custom image source.

On the regular one we do the same mapping but use a bare <img> tag. In these we don't create a thumb_id because the images will be these, but on the optimized ones in order to account correctly for the total size of files being loaded we need to differentiate between the thumbnails retrieved from IE and the previews. We’re using state.force_load to control the rendering, this is a walk-around for a glitch that happens on first load where sometimes the thumbnails won’t trigger the loaded handler.

Helper functions

For the helper functions, create a directory named js, and inside it a file named utilities.js:

export const NON_OPTIMIZED_ADDRESS = process.env.NON_OPTIMIZED_ADDRESS;

export async function handle_image(image_type, img, selected_image_url, dispatch) {

    let head = await fetch(img.src, {method: "head"}),
        size = head.headers.get("content-length");

    if (size) {
      dispatch({type: "add_image_size", image_type: image_type, image: selected_image_url, size: parseInt(size)});
    }
}

export function get_size(state, type, url) {
    let size = state[build_type_key(type)][url];
    return  size || size === 0 ? display_size(size) : "No info";
};

export function calc_total_sizes(state, type) {
    let total_size = Object.values(state[build_type_key(type)]).reduce((acc, size) => acc + size, 0);
    return display_size(total_size);
};

export function build_type_key(type) {
    return `loaded_images_${type}`;
};

function display_size(size) {
    if (size > 1048576) {
      return (Math.round((size / 1048576 + Number.EPSILON) * 100) / 100) + "MB";
    } else if (size > 1024) {
      return (Math.round((size / 1024 + Number.EPSILON) * 100) / 100) + "Kb";
    } else {
      return size + "b";
    }
};

Let's go through each element here:

• NON_OPTIMIZED_ADDRESS just proxies the value we will set as an ENV variable, to be picked from our next.config.js we created earlier.
• handle_image is the function we're using when an image finishes loading. It takes 4 arguments:
  • image_type - if it's ie or regular so we can distinguish between which kind of image was loaded
  • img - which is the html img element itself
  • selected_image_url - which is the key under which we to correlate with this image
  • dispatch - which is our reducer's dispatch function.

Now, there isn't a straightforward way in JS to get an image size that works every time for local, cross-origin, and cached files, so our solution is to issue an head request for the same resource. We use the actual img element to retrieve the exact src attribute. The response to this request includes the size of the resource, under the content-length header, but without actually transferring its content.

• get_size is an helper that retrieves the correct size, for a particular url of a given type from our state.
• calc_total_sizes sums up the sizes of all resources we have retrieved for a given type.
• build_type_key is a simple helper to return the correct state key for a given type.
• display_size is a helper that formats an integer value representing bytes into either mb, kb or b

With this in place, let's add the css for our thumbnails section, IL169 - LL236

Image Preview component

Finally, on image_preview.js:

import { get_size, calc_total_sizes } from "../js/utilities.js";

export default function ImagePreview({ state, type, prefix,  children }) {
    let prefixed = `${prefix}${state.selected_image_url}`;
    
    return (
	<div className="image-container-wrapper">
	  <div className="image-info-container">
	    <div className="image-info-size">{state.selected_image_url ? get_size(state, type, prefixed) : "No info"}</div>
	    <div className="image-info-size-total">Total: {calc_total_sizes(state, type)}</div>
	  </div>
	  <div className="image-container">
            {state.selected_image ?
              children :
              <p>Select an Image to preview</p>
	    }
	  </div>
	</div> 	    
    );
};

This one is a simple wrapper. If there's no selected image it shows Select an image to preview, and in case the size of the file hasn't been retrieved yet (it's an async function) it shows No info, if there's a selected image we use the children prop to display it.

Let's add the CSS styling for this part IL239 - LL336

Final index.js

Lastly, we're ready to finish our index.js file (omitting the reducer and initial state) :

import Head from "next/head";
import Sidebar from "../components/sidebar.js";
import ImagePreview from "../components/image_preview.js";
import Thumbnails from "../components/thumbnails.js";
import Copyright from "../components/copyright.js";

import { Image } from "@imageengine/react";

import { useReducer, useEffect } from "react";

import {
    build_type_key,
    handle_image,
    NON_OPTIMIZED_ADDRESS
} from "../js/utilities.js";


export default function Home() {
    const [state, dispatch] = useReducer(settings_reducer, initial_settings_state);

    useEffect(() => {
	setTimeout(() => dispatch({type: "force_load", value: true}), 25)
    }, []);
    
    return (
	<div className="main-container">
	  <Head>
	    <title>ImageEngine Optimized Assets</title>
	  </Head>
	  <Sidebar state={state} dispatch={dispatch} />
	  <div className="main-area">
	    <h2 className="page-title">ImageEngine (left) vs Unoptimized (right)</h2>
	    <Copyright state={state} dispatch={dispatch} />
	    <div className="main-area-images">
              <ImagePreview state={state} type="ie" prefix={""}>
                <Image
                  src={state.selected_image_url}
                  directives={{
                    outputFormat: state.file_type,
                    height: state.height,
                    width: state.width,
                    fitMethod: state.fit_type
                  }}
                  onLoad={(event) => handle_image("ie", event.target, state.selected_image_url, dispatch)}
                />
              </ImagePreview>
              <ImagePreview state={state} type="regular" prefix={NON_OPTIMIZED_ADDRESS}>
                <img src={NON_OPTIMIZED_ADDRESS + state.selected_image_url} onLoad={(event) => handle_image("regular", event.target, (NON_OPTIMIZED_ADDRESS + state.selected_image_url), dispatch)}/>
              </ImagePreview>
	    </div>
	    <Thumbnails state={state} dispatch={dispatch} />
	  </div>
	</div>
    );
};

Besides the useReducer we also imported useState and used it here:

  useEffect(() => {
    	setTimeout(() => dispatch({type: "force_load", value: true}), 25)
    }, []);

To show the thumbnails so their onload handle triggers properly.

On the first <ImagePreview> component we pass as a child an ImageEngine Image component, and we set both the src prop to our selected_image_url and the directives prop to an object containing the customisable properties we have in our sidebar. We could add further directives here. Because we wrapped our _app.js content with an <ImageEngineProvider> it will automatically be used by the <Image> component. In a normal situation we might want to debounce the changes to our reducer’s state so it doesn’t immediately request a new image.

In the second <ImagePreview> instead we pass a simple html <img> tag.

If you try to see the website it should be working, but the images won't be loading as we haven't them yet - you can download them from images and place them in public/images. We aren't using yet an actual IE distribution, so we won't have any differences in sizes, except that the IE side will count each image twice as you pick them, once for the thumbnails, another for the previews - as we discussed earlier.

If you want to add responsiveness to the layout, there’s a few more lines of css to add, IL338 - LL372.

Bucket, IE distribution, static build and deploying

Now that we have our app ready, let's set up an AWS S3 bucket to host it. In this case we will host it bare on S3, as a static website, making the whole bucket public, but outside of a tutorial, this bucket shouldn't be public at all and instead use cloudfront in front (get it?). You can do it from the aws console interface, or using the aws cli. With the cli, change the bucket-name and use-west-2 region to something you want on the following commands:

aws s3 mb s3://bucket-name --region us-west-2 aws s3 website s3://bucket-name/ --index-document index.html --error-document 404.html

Because we created the bucket in us-west-2 region and set it as a static website, it will be hosted on the following url http://bucket-name.s3-website-us-west-2.amazonaws.com

If using AWS Console just create a new bucket and set it as a static website after it's created (from the Properties tab),

Now that we have an host we can refer to, let's create an ImageEngine distribution for it:

imageengine.io

changing the url address used in the video to that of your bucket. Then from your account panel, select Edit Engine and on the CORS Support section change the HTTP methods allowed value from GET, to GET,HEAD.



And that’s all for IE. Right now we don't have any files on our bucket so we don't have a website yet. Next.js allows you to build a static website that we can use in S3. Since we have the url for our bucket and for our IE distribution we can build our website using those values.

Let's add a specific command to our package.json that does it for us, under the scripts key, add "static": "next build && next export". Save the file. From the command line we can now run (substituting the values for the ones you have):

DELIVERY_ADDRESS="https://your-image-engine.cdn.imgeng.in" NON_OPTIMIZED_ADDRESS="http://bucket-name.s3-website-us-west-2.amazonaws.com" npm run static

This will create all necessary files for a static website in the out directory of your app dir. The last step is to upload those files into our bucket. We can do that manually through the console interface, or using the cli:

aws s3 cp out s3://bucket-name/ --recursive --acl public-read

Notice we need to set the Access Control List to public-read. If doing the upload through the interface we would need to set it on the upload panel, before uploading them.

And that's it, maybe give it a minute and you should be able to visit http://bucket-name.s3-website-us-west-2.amazonaws.com and see it working!

Now that you have the ImageEngine distribution available you can also run it locally with DELIVERY_ADDRESS="https://your-image-engine.cdn.imgeng.in" npm run dev.

If you omit the DELIVERY_ADDRESS then it will serve the local assets for both parts as it did initially. This is the same if you're integrating ImageEngine into an existing website, you can do all the code changes to use the Image component, while still relying on the normal image urls, and once ready, build a version with the correct DELIVERY_ADDRESS (or if running on a nodejs server restart it with the proper env set). If you’re not using Nextjs, or need some more control on your image components, you can also build the query string for the directives and use it directly in any img element’s src, or in http requests.

Conclusion

It's easy to add IE to a website or when starting a new project, as it involves just attaching a query string parameter to your image src's. In the same way it's also easy to leverage new HTML image attributes, such as srcset/sizes to define particular viewport dimensions with specific optimisations, as again, it just requires a query string.

If you're serving a significant amount of images and you don't have an image optimisation pipeline in place, need finer grain control than things like Next.js image optimisations, aren't running a Nodejs web server, or don't want to worry about that part of your infrastructure, then IE makes a lot of sense - its CDN is really fast as is their on-the-fly transformation engine - on top of that given their work on adjacent areas - device display identification - any improvements to their algorithms are automatically applied to your existing sources, and new options when available easily integrated.

]]> https://micaelnussbaumer.com/posts/writing-a-html-server-in-elixir/ Sat, 14 Sep 2019 16:55:26 UTC https://micaelnussbaumer.com/posts/writing-a-html-server-in-elixir/ Building a basic http server in Elixir.

I've been writing an http server in Erlang, which is still far from finished but coming together slowly, as a way of learning more in depth the http protocol and how to deal with it from scratch.

To write down my findings and share them I decided I should write a post on how to do that and in the process show some very neat more lowlevel aspects of erlang. There's a good range of http server libs available in Erlang, if you use Phoenix you'll be using Cowboy underneath, each one with its own design decisions, all of them pretty fine.

We won't be tackling in this post websockets nor any proper http connection upgrades, http2 and such, that will be hopefully for future posts, but instead we'll see how to create a simple socket acceptor pool, use gen_statems to create a way to run a "pipeline", design a basic router, parse the incoming requests, etc.

You can find the final source code for this at satellitex repo up to the end of the tutorial.

We'll have at least 4 components:

• the socket acceptor pool - creates a number of ready to work socket listeners waiting for requests on a given port
• the gen_statem that will use a modular and exchangeable pipeline definition to parse the request and route it appropriately
• the router module that allows a custom DSL that generates a proper router for our gen_statem to call with the appropriate routes and data
• a response creator, to send the responses to our clients when they visit our website

We'll also need several parsing functions.

To follow along you need to have installed erlang 22 and elixir 1.9 (compiled with erlang 22). After having those installed, lets start by creating an umbrella app, from the command line:

mix new satellitex --umbrella

Lets go into our newly created directory and further into the apps folder and create our socket acceptor app (it will be a simple lib and not an application with a supervision tree):

cd satellitex/apps
mix new launchpad 

Now we'll be using gen_statem and so we should add some key properties to our logger configuration so that we can get system reports if the gen_statem crashes, otherwise we won't see the actual errors caused by it since Logger by default doesn't output system logs unless told to do so and gen_statem outputs its errors as system reports. Lets open our config.exs file for the umbrella (satellitex/config/config.exs) and make it look like the following:

use Mix.Config

import_config "../apps/*/config/config.exs"

config :logger,
  handle_otp_reports: true,
  handle_sasl_reports: true

Now save the file. This is not to say that our code will have errors, that's quite outrageous to even think, but just in case we mistyped something.

Let's start writing our acceptor. This will be our "server" per se. It won't deal with requests by itself, instead, what it will do is open a socket in a given port and then create any number of ready to fire acceptors, listening on the socket. Each acceptor will be itself an independent process, to which we give "control" of the request as soon as a tcp connection is open, so that they can read from the socket and parse the request into something structured and usable for our application. We'll dive into that shortly after, but the idea is that it will be able to accept a "pipeline" definition and process the request according to that pipeline so that it's easily extendable.

First, if you never used the gen_statem OTP behaviour it's a fairly useful abstraction. It's similar to gen_servers, but it's newer and requires a bit more ceremony. gen_servers, due to having been there for so long and having a very simple layout are used much more often but lately I've been finding that gen_statems are really nice to use in a lot of different situations, even if you're not writing anything that is a complex state machine, whenever you may have state transitions it fits very well (I already used continue handles a lot in gen_servers and now I think they're the poor man's version of a gen_statem).

They're a behaviour, so we need to declare that, and they can operate in two different modes of callback, state_functions or handle_event_functions. We'll use handle_events mode, but to just cover the differences, in handle event mode you write every callback as def handle_event(........) end and the state can be as complex as you want. In state_function mode you write functions where the name of the function is the state to which it refers and the state must always be an atom. So if you had a state of starting, you would have a def starting(.....) end function(s) to handle all events when the state machine is in that state and other similarly named functions for each state you have. I personally like better the handle mode.

Our ideal server will accept a bunch of settings to decide how to start itself, it will have a start_link function that receives these options, then call a init function with these options that opens up the socket and starts the acceptors.

So let's go to our launchpad application, and open /apps/launchpad/lib/launchpad.ex and replace it with the following:

defmodule Launchpad do
    @behaviour :gen_statem

    defstruct [:socket, :config, max_pool: 5, pool_count: 0, acceptors: %{}]
   
    def start_link(opts) do
        name = Map.get(opts, :name, {:local, __MODULE__})
    
        :gen_statem.start_link(name, __MODULE__, opts, [])
    end

    @impl true
    def callback_mode(), do: :handle_event_function

    @impl true
    def init(opts) do
        Process.flag(:trap_exit, true)
    
        port = Map.get(opts, :port, 4000)
    
        {:ok, socket} = :gen_tcp.listen(port, [:binary, {:packet, :raw}, {:active, false}, {:reuseaddr, true}])

    data = %__MODULE__{socket: socket, config: opts}
    
        {:ok, :starting, data, [{:next_event, :internal, :create_listener}]}
    end

    @impl true
    def handle_event(:internal, :create_listener, _state,
        %__MODULE__{
            socket: socket,
            config: config,
            max_pool: max,
            pool_count: pc,
            acceptors: acceptors
        } = data
    ) when pc < max do
    
        {:ok, pid} = Satellite.start_link(socket, Map.put(config, :number, pc))
        n_acceptors = Map.put(acceptors, pid, true)

    {:keep_state, %{data | pool_count: pc + 1, acceptors: n_acceptors}, [{:next_event, :internal, :create_listener}]}
    end

    def handle_event(:internal, :create_listener, :starting, data), do: {:next_state, :running, data, []}

    def handle_event(:internal, :create_listener, _state, _data), do: {:keep_state_and_data, []}

    def handle_event(:info, {:EXIT, pid, _reason}, _, %{pool_count: pc, acceptors: acceptors} = data) when :erlang.is_map_key(pid, acceptors) do
        {_, n_acceptors} = Map.pop(acceptors, pid)
        {:keep_state, %{data | pool_count: pc - 1, acceptors: n_acceptors}, [{:next_event, :internal, :create_listener}]}
    end

    def handle_event(:info, {:EXIT, pid, reason}, _, _data) do
        IO.puts("Received exit from unknown process #{inspect pid} with reason #{reason}")	
        {:keep_state_and_data, []}
    end

end

There's quite a lot going on there so let's break it up:

• @behaviour :gen_statem - we're simply stating that this module will implement the :gen_statem OTP behaviour
• defstruct .... - a structure we created to hold the relevant information about our server, it has a few keys, :socket, which will hold the socket ref once we open it, :config, which will store the configuration passed to it, :max_pool, which is the maximum number of acceptors we want to have running at each time, :pool_count, just a counter that holds how many acceptors we have running to decide if we need to start more or not, and :acceptors were we keep a map of our acceptors pids
• def callback_mode() - it's a required function for this behaviour and it informs what type of callbacks our :gen_statem will be running, in this case we set it to return :handle_event_function

The others are regular things, a start_link/1 function that takes an argument (which will be a map/struct with configuration) a init/1 function that the behaviour requires to start, in our case the argument will be the same as passed down to start_link, and then two handle_event functions, these are functions defining the behaviour of our :gen_statem and how it should act when there's either a state change or an event trigger.

In the start_link/1, for now, we will just try to get a key :name from it in order to register our process. In case no name is provided we default to {:local, __MODULE__}, which will register our process locally, on start, as Launchpad (that's what __MODULE__ gets translated to).

So for :gen_statem.start_link(name, __MODULE__, opts, []), the first argument is the name to register, the second is the module implementing the statem (in this case it's this module itself), the 3rd is the arguments passed to init/1 and the 4th is a list of keyword options for the :gen_statem initialization itself, that we're leaving empty. We could also make gen_statem anonymous by excluding the 1st argument.

On the init/1 function we're setting the process to trap exits by using Process.flag(:trap_exit, true), the reason for this is because we'll be starting each acceptor as a link, so that if our "launchpad" process dies they are killed as well, but in turn, because links are bi-directional, this makes it so that if any of the acceptors dies our "launchpad" would too. Since we set it to trap exits, instead, if an acceptor dies, what will happen is we will receive a message telling us about it, this will allow us to "know" when one of them goes down in case we want to start another one to replace it, instead of it simply exiting automatically.

Then we read the port from the config or default to 4000 and next we start the actual socket process listening on that port, with a bunch of options through :gen_tcp.listen.

The options we pass are :binary, telling we want to read the socket as binary segments, we set {:active, false}, this means that we don't want to read data immediately whenever it arrives, we want to control how and when the reading is going to happen. :reuseaddr is for the socket to allow reusing the same address and {:packet, :raw}, tells to pass the data as read. If the :gen_tcp.listen succeeds it means we were able to bind to that socket port, otherwise this will raise an error, crash, and if the user of our server started it from a supervisor hopefully crash their app, so that they notice something is wrong.

Then we just create a struct for our data, with the previously opened socket and the config map passed to it.

The init/1 callback has to return when successful an {:ok, ...statem_definitions} tuple, in our case:

{:ok, :starting, data, [{:next_event, :internal, :create_listener}]}

Means, set the internal initial state to :starting and the statem data as the contents of data. The last element is a list of actions specific to the :gen_statem behaviour that dictate any event it should trigger upon finishing this callback. In this case we say, we want to trigger a new event, a specific internal one that we can be sure is only triggered from our gen_statem, with the name :create_listener.

Hence the next handle_event functions we wrote. Each handle_event function receives as its arguments:

type_of_event, event, currrent_state, current_data

The type of event can be :internal, :info, :call, :cast, :timeout, etc... You can find all the documentation for gen_statem in erlang documentation The event is the contents of that event, and the others are self explanatory. An :internal event type can only be issued by your own code inside the gen_statem where they happen, you can be sure these were generated by you.

On the first handle_event we pattern match on our struct specifically so we can access the value of :max_pool and :pool_count. This way we can know if the maximum number of acceptors are already running or not, and we do that by setting a guard clause, when pc < max.

So if the pool_count is lower than the max we have set, then open a new one, otherwise the next function head that matches will be the one called - we have two, one if we're in the :starting state, which will just switch to the state :running while keeping everything else, and another one that matches whenever the state is not starting and the max acceptors have already been created, it does nothing by returning {:keep_state_and_data, []}, which is a shorthand for {:next_state, :same_state, same_data, []}.

When we create a new listener, we increment the pool count, we add that pid as a key to our acceptors map and we trigger the same event again. So it repeats the listener creation until we have the max number of acceptors created and then sits waiting, doing a siesta.

Lastly we have a function to receive the EXIT message in case any of our linked processes dies, because we're trapping exits. When we receive one we decrease the pool count by one and trigger a new event to create another listener. We know this exit is from an "acceptor" process because we use the guard :erlang.is_map_key(acceptors, pid), basically saying only match this function when the pid on the EXIT message is a key on the map acceptors, meaning one of the pids from an acceptor we started (that we know because we placed them in the acceptors map as keys). If it's not we do a naive log of it because we should theoretically not have received that exit signal.

So our app is actually a library, this is because we don't want to control how it's started from itself, instead we expose the start_link/1 function that then someone can start on their own supervisor trees. We'll see that later but, for now, lets see if it does something already. Go to the root of the umbrella and start an iex shell, with:

iex -S mix run

You should see some warnings, and:

Erlang/OTP 22 [erts-10.4.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

==> launchpad
Compiling 1 file (.ex)
warning: function Satellite.start_link/2 is undefined (module Satellite is not available)
  lib/launchpad.ex:39


19:02:57.776 [info]  Application logger started at :nonode@nohost
 
19:02:57.776 [info]  Application launchpad started at :nonode@nohost

We can see already a problem, we don't have a Satellite.start_link/2 function neither a Satellite module, and if we try to start our Launchpad we see that error actually happening:

iex(1)> Launchpad.start_link(%{})
{:ok, #PID<0.151.0>}
iex(2)> 
19:05:14.065 [error] Process Launchpad (#PID<0.151.0>) terminating
** (UndefinedFunctionError) function Satellite.start_link/2 is undefined (module Satellite is not available)
    Satellite.start_link(#Port<0.8>, %{number: 0})
    .....

But the Launchpad did start and effectively tried to start one acceptor. Lets write the first completely barebones version of the acceptor.

Lets navigate again to apps/ dir, and do:

mix new satellite

Now let's open apps/satellite/lib/satellite.ex

defmodule Satellite do
  @behaviour :gen_statem
  
  defstruct [:socket, :config, :conn, pipeline: [], request: %{}]
   
  def start_link(socket, opts) do
    :gen_statem.start_link(__MODULE__, {socket, opts}, [])
  end

  @impl true
  def callback_mode(), do: :handle_event_function
  
  @impl true
  def init({socket, config}) do
    {:ok, :waiting,  %__MODULE__{socket: socket, config: config}, [{:next_event, :internal, :wait}]}
  end

  @impl true
  def handle_event(:internal, :wait, :waiting, %{socket: socket} = data) do
    {:ok, conn} = :gen_tcp.accept(socket)
    
    :gen_tcp.controlling_process(conn, self())

    {:next_state, :parsing, %{data | conn: conn}, [{:next_event, :internal, :read}]}
  end

  def handle_event(:internal, :read, :parsing, %{conn: conn} = data) do
    case :gen_tcp.recv(conn, 0, 1000) do
      {:ok, packet} ->
        IO.inspect(packet, label: "received")
        n_data = %{data | request: packet}
        {:next_state, :response, n_data, [{:next_event, :internal, :send_response}]}

      {:error, reason} ->
        IO.inspect(reason, label: "error on recv")
        {:keep_state, data, [{:next_event, :internal, :close}]}
    end
  end

  def handle_event(:internal, :send_response, :response, %{conn: conn, request: request} = data) do
     b = :erlang.iolist_to_binary(request)
     response = :io_lib.fwrite(
       "HTTP/1.0 200 OK\nContent-Type: text/html\nContent-Length: ~p\n\n~s",
       [:erlang.size(b), b]
     )

     :gen_tcp.send(conn, response)

     {:keep_state, data, [{:next_event, :internal, :close}]}
  end

  def handle_event(:internal, :close, _, %{conn: conn} = data) do
    :gen_tcp.close(conn)
    {:next_state, :waiting, %{data | conn: nil, request: %{}}, [{:next_event, :internal, :wait}]}
  end

end

This is nowhere close to what we want, but it will allow us to already receive http requests. Lets see that and then go through the code. Go to the root of your umbrella on the terminal and again run iex -S mix run

Inside the shell now do:

Launchpad.start_link(%{})

You should see {:ok, #PID....}

Now open a browser and visit http://localhost:4000.

Voilá, we have a "server" listening on port 4000. It doesn't do really anything, it just echoes whatever was the request as a text/html response.

To confirm that indeed we have 5 acceptors currently running, let's change this line:

def handle_event(:internal, :read, :parsing, %{conn: conn} = data) do
    case :gen_tcp.recv(conn, 0, 1000) do
      {:ok, packet} ->
        IO.inspect(packet, label: "received")
        n_data = %{data | request: packet}
        {:next_state, :response, n_data, [{:next_event, :internal, :send_response}]}

      {:error, reason} ->
        IO.inspect(reason, label: "error on recv")
        {:keep_state, data, [{:next_event, :internal, :close}]}
    end
end

To:

def handle_event(:internal, :read, :parsing, %{conn: conn, config: %{number: number}} = data) do
    case :gen_tcp.recv(conn, 0, 1000) do
      {:ok, packet} ->
        IO.inspect(packet, label: "received")
        n_data = %{data | request: packet <> "\n\nAcceptor number: #{number}"}
        {:next_state, :response, n_data, [{:next_event, :internal, :send_response}]}

      {:error, reason} ->
        IO.inspect(reason, label: "error on recv")
        {:keep_state, data, [{:next_event, :internal, :close}]}
    end
end

Save the file and on the iex shell run recompile().

Now visit the browser and refresh the page some times. You should see the acceptor changing as you do. Which means that each request you're sending from the browser is being answered by a different process altogether.

If you look in the shell you should also be seeing the output from the IO.inspect functions we placed.

So this string you're seeing will vary depending on the browser, but it will have a structure that is the same. Using Safari I see this:

GET / HTTP/1.1\r\nHost: localhost:4000\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36\r\nSec-Fetch-Mode: navigate\r\nSec-Fetch-User: ?1\r\nDNT: 1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3\r\nSec-Fetch-Site: none\r\nAccept-Encoding: gzip, deflate, br\r\nAccept-Language: pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7\r\n\r\n"
received: "GET /favicon.ico HTTP/1.1\r\nHost: localhost:4000\r\nConnection: keep-alive\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nSec-Fetch-Mode: no-cors\r\nDNT: 1\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36\r\nAccept: image/webp,image/apng,image/*,*/*;q=0.8\r\nSec-Fetch-Site: same-origin\r\nReferer: http://localhost:4000/\r\nAccept-Encoding: gzip, deflate, br\r\nAccept-Language: pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7\r\n\r\n

There's a whole bunch of documents detailing exactly how an HTTP request (on the various versions of the protocol) must be constructed, what it can contain, how those things are encoded and the same for the HTTP responses. We'll not read the RFC for the HTTP Protocol but basically implement the logic for parsing these requests and deliver a response on a more lightway approach.

We can see that the request is basically:

VERB whitespace PATH whitespace PROTOCOL/VERSION\r\nHEADERS\r\n\r\n

where \r\n is the carriage return escape sequence.

After the headers there would be a body but in this case since it's a get request there's none. In case of having a body the request should include a Content-Length header stating the size of the body although it's not totally required that it has.

So if we approach this from the point of view of creating a pipeline, basically we would need to:

• Parse the Verb
• Parse the Path
• Parse the Protocol
• Parse the headers
• Maybe Parse the Body if it has one
• Dispatch to the appropriate route

On each step we can then store the information we parsed so that we can later decide how to respond to it, which we'll do by dispatching that request into an appropriate handler.

We should read the RFC to make sure our server is completely correct, but like we said before, this is a lightweight version so we'll just read the basics from that request to decide what we need.

Verb, Path and Protocol are each one separated by a white space. Headers are separated from the previous section and between them by \r\n, meaning carriage return, a set of two characters, \r return and \n newline, each header key ends with : followed by a white space, after which comes the respective value.

The end of the headers is separated by double carriage return, \r\n\r\n. After this is the body and this can be formatted in different ways depending on the type of request.

There's many ways we can implement the parsing part, but we want it to be modular, so that anybody can hook their own things into a pipeline if they want, or change it completely, the idea is the basic pipeline will parse everything needed and then dispatch the request. We will want to implement some control on maximum length of the requests (or allow that even if by default it accepts anything). We want the pipeline to be halt-able, meaning that we can trigger an halt between any actions. We don't want to have explicit hardcoded behaviour, we want each step to be a call to a function that can be customised.

We will also want to have different kinds of actions, we'll want to have parsing/reading actions, that get part of the data from the socket and do something with it (parse it and if needed store it in the context of that specific request), we want to have actions that simply change the form of the data according to something in a certain place when they occur in the pipeline and we want actions that check the current state of what we have parsed/stored to decide if to continue parsing, halt, error, or dispatch.

One problem we will always run into is that the HTTP protocol doesn't require a request to say how many bytes it is in total, so we can't know how many bytes to read from the socket until we figure the end of the request. To add to that, we have the fact that we might read incomplete requests due to the buffering on the socket at the os level (and subsequent transmission to the Erlang VM). Most of the times the socket when read will contain all the data, but on big length requests we might need to read more than once from it.

When we set the socket to {:active, false} we explicitly told that we will somehow read the socket by ourselves. The reason to do this is because if we say {:active, true} we forego any possibility of applying backpressure to the data being sent and read from the socket, and someone could potentially flood our server with gigantic requests, possibly crashing it down in the process.

By saying {:active, false} the BEAM will not automatically send that data, it will be in the socket buffer, waiting to be read at which point we need to manually request a reading on it. One normal way of doing it is after we set the acceptor and pass the controlling process to the one that will receive the request, to set the socket to {:active, :once}, which means it will process once the data currently in the buffer (emptying it so at the os level the client tcp connection can push more data into it), send it as a message to the controlling process, and then switch automatically to {:active, false} again.

This has the benefit of allowing us to apply backpressure, it's like opening a faucet once for a little while and immediately closing it again. If we need to read more we repeat the same.

Other option is by using :gen_tcp.recv. In this way you can control how many bytes to read along with a timeout for it and instead of receiving that data as a message, you get the result of the read as the return of the function with {:ok, data}, or an {:error, reason} if there was some error (the socket was closed on the client for instance), or :timeout if the amount of bytes specified couldn't be read in that given time frame. :gen_tcp.recv/3 takes the accepted socket from where to read as its first argument, the number of bytes to read as its second argument and the timeout as its third argument.

One thing to notice is that, if we specify a number that is not zero, then this function will only return after having read EXACTLY that number of bytes. In this case, this is not very useful, because we don't know beforehand how many bytes there will be in total, so unless we read a single byte at a time we would always end up timing out. Reading one byte at a time is not efficient in any way either. But if we specify the number as zero (0) then recv will act like {:active, :once}, in that it reads the whole available buffer in one go, and then returns that data. If we need to read more we call again recv and again we get whatever is in the buffer up to that point and so on. This again allows us to manually control the backpressure and the flow of data (we can even when creating the socket say what's the size we want for our buffer and other things but we'll not cover that).

(sidenote: you might ask then why is there an option to specify the number of bytes and not only 0, and the reason is, if you control the client (it's an application you wrote) or you have a defined protocol where you can specify the length, then you can optimise the readout to your specific system, network and even the client application flow. Of course when dealing with HTTP requests, that's not the case)

We can then implement a way of limiting the maximum number of bytes a request might have before we close the connection. We'll see that in more detail when we get to it. Now lets start implementing our basic request parser.

First we'll need to define a structure that will hold the parsed information in an organised way, so that we can check it. We also need to define a structure to hold the configuration information.

For the request lets create another module in the satellite app, apps/satellite/lib/satellite_request.ex . Open the file and add to it:

defmodule Satellite.Request do

  defstruct [
    :verb,
    :protocol,
    :version,
    :accept,
    :body,
    path: [],
    host: [],
    query: %{},
    headers: %{},
    finished_headers: false,
    halt: false,
    params: %{},    
 ]
  
end

The :query, :headers and :params field are maps, because they're key-value elements. :path and :host are defined as lists although the url requested and the host will be strings if we want to dispatch on certain paths, match parts of the path, etc, and the same for the host/domain/port, we will need to decompose it into individual elements in order to do so. The remaining keys will either be booleans or strings/atoms that we will fill as we move through the pipeline.

Save that file.

Now let's create one for the configuration. The previous request structure is only used by the acceptors gen_statem and possibly any lib that depends on our whole application. On the other hand the config is used by the launchpad and satellite. So we will create a new app/lib to store these shared elements. We haven't added any dependencies to our apps until now but we will afterwards.

Go to the apps folder on the terminal again, and execute mix new satellite_shared.

Now create and open a file at apps/satellite_shared/lib/satellite_configuration.ex

Place inside it the following:

defmodule Satellite.Configuration do

  defstruct [
    :pipeline,
    :name,
    :router,
    port: 4000,
    acceptors: 5,
    max_size: 50_000,
    keep_full_request: false,
   
 ]

end

Now open apps/launchpad/mix.exs and replace the deps function by:

defp deps do
    [
      {:satellite, in_umbrella: true},
      {:satellite_shared, in_umbrella: true}
    ]
end

Open the apps/satellite/mix.exs and replace the deps function by:

defp deps do
    [
      {:satellite_shared, in_umbrella: true}
    ]
end

The reason we were able to run previously without specifying these dependencies is because mix will compile and load all applications and modules in the umbrella, but if we were to assemble a release, then the compilation would fail, because the dependencies weren't specified.

Basically we're now saying launchpad depends on both the app satellite and satellite_shared and the app satellite depends on satellite_shared.

Ok, so now let's think about how we want to actually have this working. (I've done the thinking for you but I'll pretend it's fresh).

I'll throw in that one neat way of having it defined is as a list of things, so that we can move through it as we go. Each element of this list should have the type of action it is, the name for that particular step, which function to call for doing the step and also, because we might get things in several receives, an accumulator specific for that step. So say we want to encode the step for parsing the verb out of the request, given the 4 elements we mentioned previously, it could look like this:

{:read, :verb, &Satellite.Verb.parse/3, <<>>}

So the action would be of the type :read, the name of the step would be :verb, the function that would execute this step would be Satellite.Verb.parse/3 and the accumulator would be an empty binary.

Let's start writing it, first create another file in the satellite lib, apps/satellite/lib/satellite_defaults.ex and write in it:

defmodule Satellite.Defaults do
  
  def default_pipeline() do
    [
      {:read, :verb, &Satellite.Verb.parse/3, <<>>}
    ]
  end
end

Save it. We'll add the others as we go. Now we need to create a module Satellite.Verb. So lets create another file apps/satellite/lib/satellite_verb.ex and put in it:

defmodule Satellite.Verb do

  def parse(ctx, rem, <<"GET">>), do: {:done, %{ctx | verb: :get}, rem}

  def parse(ctx, rem, <<"POST">>), do: {:done, %{ctx | verb: :post}, rem}

  def parse(ctx, <<>>, acc), do: {:cont, ctx, acc}

  def parse(ctx, <<?\s, rem::bits>>, acc), do: parse(ctx, rem, acc)

  def parse(ctx, <<h, rem::bits>>, acc), do: parse(ctx, rem, <<acc::bits, h>>)

  def parse(ctx, _, _), do: {:error, "Parsing Verb"}
  
end

Here the idea is very simple, we create a recursive function. First this function will be called the first time with (ctx, the_request_binary_from_the_socket, <<>>) because the initial accumulator is <<>> as set in our pipeline definition. So it won't match the first 4 functions, but will match the 5th, were we pluck 1 single character from it, add call again parse with the remaining of the packet, while adding that single character we removed to the accumulator.

It repeats this until the accumulator is <<"GET">> or <<"POST">>, at which point we consider it done because we have a matching verb (right now it will fail for any other verb). If there was a white-space somewhere, it would ignore it and move on to the next character, but because it will assemble GET or POST before that it's never called, anyway, I'm leaving it there just because.

Most parsing will be similar to this, but it will have its own peculiarities. This should work even if for instance we were reading 1byte at a time from the socket, because in that case, we would read the first, rendering an accumulator of <<"G">>, and an remaining of <<>>, which would match the 3rd function and in that case return {:cont, ctx, acc}, which would trigger a new read from the socket and consequent parsing.

Now how do we make our satellite actually work with this? So one thing we can do is treat our state of the gen_statem as the current step in the pipeline. With pattern matching we can describe this very elegantly, because if the pipeline is a list, we can simply pluck the first element from it and store it as the current state, and every time it moves we pluck another one, until we have no more steps to do, at which point we know we've finished the pipeline and parsed the full request. The complete story is a bit different because of the specifics of http requests, but that's the overall idea, we'll adjust as we need while going through it. So now we need to change how the launchpad is started, so that it uses our recent Satellite.Configuration struct.

Add this to apps/launchpad/lib/launchpad.ex

  alias Satellite.Configuration, as: Config


  defp set_defaults(config) do
    maybe_set_name(config)
    |> maybe_set_port()
    |> maybe_set_pipeline()
  end
  
  defp maybe_set_name(%Config{name: name} = config), do: maybe_set(:name, name, {:local, __MODULE__}, config)

  defp maybe_set_port(%Config{port: port} = config), do: maybe_set(:port, port, 4000, config)

  defp maybe_set_pipeline(%Config{pipeline: pipeline} = config) do
	maybe_set(:pipeline, pipeline, Satellite.Defaults.default_pipeline(), config)
  end

  defp maybe_set(key, nil, default, config), do: Map.put(config, key, default)
  defp maybe_set(_, _, _, config), do: config

And change start_link and init to the following:

def start_link(%Config{} = config) do
    %{name: name} = ok_config = set_defaults(config)
    
    :gen_statem.start_link(name, __MODULE__, ok_config, [])
end

def init(%{port: port} = config) do
    Process.flag(:trap_exit, true)
    
    {:ok, socket} = :gen_tcp.listen(port, [:binary, {:packet, :raw}, {:active, false}, {:reuseaddr, true}])

    data = %__MODULE__{socket: socket, config: config}
    
    {:ok, :starting, data, [{:next_event, :internal, :create_listener}]}
end

We created some helper functions to set defaults in case it isn't passed into and also changed the start_link and init to use that. If there's no :name, :port or :pipeline set in the config it will set them to some default. Because we're now matching explicitly on the %Config{} struct in start_link/1, if you pass anything else it will fail with an unmatched function call.

Now we need to change the satellite.ex file, to use the pipeline and create the handle_event functions that will coordinate that.

I'm going to past the full file as it should look and we go through the changes:

defmodule Satellite do
  @behaviour :gen_statem

  alias Satellite.Request

  defstruct [:socket, :config, :conn, pipeline: [], request: %Request{}]
  
  @impl true
  def callback_mode(), do: :handle_event_function
  
  def start_link(socket, opts) do
    :gen_statem.start_link(__MODULE__, {socket, opts}, [])
  end
  
  @impl true
  def init({socket, config}) do
    {:ok, :waiting,  %__MODULE__{socket: socket, config: config}, [{:next_event, :internal, :wait}]}
  end

  @impl true
  def handle_event(:internal, :wait, :waiting, %{socket: socket, config: %{pipeline: pipeline}} = data) do
    {:ok, conn} = :gen_tcp.accept(socket)
    
    :gen_tcp.controlling_process(conn, self())

    n_data = %{data | conn: conn, pipeline: pipeline, request: %Request{}}

    set_next_step(n_data, <<>>)
 end

  def handle_event(:internal, :read, _, %{conn: conn} = data) do
    case :gen_tcp.recv(conn, 0, 1000) do
      {:ok, packet} ->
        {:keep_state_and_data, [{:next_event, :internal, {:parse, packet}}]}

      {:error, reason} ->
        {:next_state, :response, %{data | request: "Error: #{inspect reason}"}, [{:next_event, :internal, :send_response}]}
    end
  end

  def handle_event(:internal, {:parse, packet}, {type, name, fun, acc}, %{request: request} = data) do

    case fun.(request, packet, acc) do
      {:cont, n_request, n_acc} ->
        {:next_state, {type, name, fun, n_acc}, %{data | request: n_request}, [{:next_event, :internal, :read}]}

      {:done, n_request, remaining} ->
        set_next_step(%{data | request: n_request}, remaining)

      {:error, reason} ->
        {:next_state, :response, %{data | request: "#{inspect reason}"}, [{:next_event, :internal, :send_response}]}
    end
    
  end

  def handle_event(:internal, :send_response, :response, %{conn: conn, request: request} = data) do
     
     :gen_tcp.send(conn, make_response(request))

     {:keep_state, data, [{:next_event, :internal, :close}]}
  end

  def handle_event(:internal, :close, _, %{conn: conn} = data) do
    :gen_tcp.close(conn)
    {:next_state, :waiting, %{data | conn: nil}, [{:next_event, :internal, :wait}]}
  end


  defp set_next_step(%{pipeline: [h | t]} = data, remaining) do

    event = case remaining do
              <<>> -> :read
              _ -> {:parse, remaining}
            end

    {:next_state, h, %{data | pipeline: t}, [{:next_event, :internal, event}]}
  end

  defp set_next_step(%{pipeline: [], request: request} = data, _remaining) do

    n_request = "#{inspect request}"
    
    {:next_state, :response, %{data | request: n_request}, [{:next_event, :internal, :send_response}]}
  end

  defp make_response(request) do
    b = :erlang.iolist_to_binary(request)
    :io_lib.fwrite(
      "HTTP/1.0 200 OK\nContent-Type: text/html\nContent-Length: ~p\n\n~s",
       [:erlang.size(b), b]
    )
  end
end

So first, we changed the defstruct to hold a %Satellite.Request{} (we aliased the module so we can write %Request{} instead) by default instead of an empty map.

Then the init is the same as before, but the :wait event while :waiting is different. Now, because we know we have the pipeline in the %Config{} structure, we take out that through pattern matching and then after having made a connection we create a new structure with the :conn, the :pipeline value and an empty %Satellite.Request{} as the :request, we then, instead of returning the statem tuple, call the set_next_step function, which takes a Satellite struct, and the bytes read from the socket as the 2nd argument. In this case, since we're just starting the pipeline we don't have any bytes, so we pass an empty binary <<>>.

This function then pattern matches on the :pipeline key and sees if we have at least one element there, and in case we do, it means we haven't finished the pipeline yet. In that matching function we also check the already read data we have available to parse and depending wether it's empty (<<>>) or not, we set the next event for the state machine. This event will either be {:parse, data} or :read in case we don't have any data to parse. First time this is called it will obviously be :read because we passed <<>>.

In case we have no other elements in the pipeline, instead, we know we have reached the end of the pipeline and as such should send a response back. This function will change of course, for now it just does an inspect on the current request struct so that it becomes a stringified version, and then sets the next event to :send_response.

On the first iteration though we have a pipeline element, we will have a single one, so the function match will look like this:

set_next_step(%{pipeline: [h | t]} = data, remaining)

Where h will be {:read, :verb, &Satellite.Verb.parse/3, <<>>} and t will be [].

Because remaining is <<>>, event will be :read. So the final return from the set_next_step will be:

{:next_state, {:read, :verb, &Satellite.Verb.parse/3, <<>>}, %{data | pipeline: []}, [{:next_event, :internal, :read}]}

This shows how cool is the :handle_event_function type of callbacks, due to that we can have complex states as this one, which allows us to describe in very clean steps what should happen when such states have to process new events.

This means that now, it will trigger the event :read that doesn't care about what state it's in, and just does what we were doing before by reading from the socket. When it reads then it sets the next event to be {:parse, data_that_was_read}, and now this will trigger the next event were we parse and - in this case - we do care about what state we're in because it contains the function we need to call to handle this step.

We had set the state to be {:read, :verb, &Satellite.Verb.parse/3, <<>>}, which means the accumulator starts empty. We also used the & notation to capture that function, because what we want is a reference to a given function so that we can then apply it. We do that by calling fun.(request, packet, acc). The first argument is the current %Request struct as it is, the second is what was read from the socket up till now, and the third is the accumulator, which starts empty.

This function should return one of 3 possible outcomes (we would write a proper behaviour for this but now we'll skip it):

• {:cont, new_request_struct, new_acc}

this return means that that given function was unable to finish with the current data retrieved from the socket, so we should ask to read more from the socket, it replaces the state of the gen_statem by almost the same as it is currently, with exception of the acc, which gets replaced by the new_acc returned from the function. It also replaces the :request key in the gen_statem's data, in case the new_request_struct has changed. When we get this event, the next event will be :read and afterwards it will trigger this event again, with new data read from the socket. The new_request_struct might or may not have changed, but to simplify logic we just replace it.

• {:done, new_request_struct, remaining}

this return means, the function was able to according to its logic finish all it had to do, so this step can be considered complete, lets move to the next one. It includes the new_request_struct and any bytes that weren't parsed. This calls set_next_step which will follow the logic we already discussed, placing the next step in the pipeline as the current. In this case we might or may not have any remaining bytes available, and set_next_step will decide accordingly what is the next event to trigger

• {:error, reason} this is self explanatory - right now it just translated whatever is the error reason into a string and jumps right to the :send_response event. We will further down change this and the :send_response steps as well

We will also need to add proper error catching but everything at its time.

If we look at what we have right now, we should be able to see some response already containing our %Satellite.Request{} struct and the proper verb parsed and included in it. Lets recompile / restart the iex shell, and do now:

Launchpad.start_link(%Satellite.Configuration{})

You should see again the familiar {:ok, pid...} Now lets open up a browser and visit again http://localhost:4000 - you should be greeted by:

%Satellite.Request{accept: nil, finished_headers: false, halt: false, headers: %{}, host: nil, params: %{}, path: [], protocol: nil, query: %{}, verb: :get, version: nil}

And as you can see, the :verb key is set to :get. Very nice very good, great success!

Let's go adding one by one the remaining parsing steps, open apps/satellite/lib/satellite_defaults.ex and replace it by:

defmodule Satellite.Defaults do
  
  def default_pipeline() do
    [
      {:read, :verb, &Satellite.Verb.parse/3, <<>>},
      {:read, :path, &Satellite.Path.parse/3, {false, [], <<>>}}
    ]
  end
  
end

Now let's create the Satellite.Path module, in apps/satellite/lib/satellite_path.ex. This one is going to be more complex because we want to separate the path into it's individual constituents, so if someone would visit http://localhost:4000/basic/path our final path list would look like ["basic", "path"] and because a path might include also query strings, in the form of ?key=value&key2=value2 we need to do more work, and in case there is query string we need to place it on the :query key of the %Request{} struct.

The reason we want to do this will become clearer when we design the dispatching mechanism, but it will allow us to do some cool stuff with it.

defmodule Satellite.Path do

  def parse(ctx, <<>>, acc), do: {:cont, ctx, acc}
  def parse(ctx, <<?\s, rem::bits>>, {false, _, _} = acc), do: parse(ctx, rem, acc)

  def parse(ctx, <<?\s, rem::bits>>, {true, acc, prior}) do
    n_acc = maybe_add_prior(acc, prior)
    
    n_ctx = %{
      ctx |
      path: Enum.reverse(
        List.flatten(n_acc)
      )
    }

    {:done, n_ctx, rem}
  end

  def parse(%{query: q} = ctx, <<?\s, rem::bits>>, {:query, :value, key, value}) do
    {:done, %{ctx | query: Map.put(q, key, value)}, rem}
  end

  def parse(ctx, <<?\s, rem::bits>>, _), do: {:done, ctx, rem}

  def parse(ctx, <<?=, rem::bits>>, {:query, :key, key}) do
    parse(ctx, rem, {:query, :value, key, <<>>})
  end

  def parse(ctx, <<h, rem::bits>>, {:query, :key, key}) do
    parse(ctx, rem, {:query, :key, <<key::binary, h>>})
  end

  def parse(%{query: q} = ctx, <<?&, rem::bits>>, {:query, :value, key, value}) do
    parse(%{ctx | query: Map.put(q, key, value)}, rem, {:query, :key, <<>>})
  end

  def parse(ctx, <<h, rem::bits>>, {:query, :value, key, value}) do
    parse(ctx, rem, {:query, :value, key, <<value::binary, h>>})
  end

  def parse(ctx, <<?/, rem::bits>>, {_, acc, prior}) do
    n_acc = maybe_add_prior(acc, prior)
    parse(ctx, rem, {true, n_acc, <<>>})
  end

  def parse(ctx, <<??, rem::bits>>, {:true, acc, prior}) do
    n_acc = maybe_add_prior(acc, prior)
    n_ctx = %{
      ctx |
      path: Enum.reverse(
        List.flatten(n_acc)
      )
    }

    parse(n_ctx, rem, {:query, :key, <<>>})
  end

  def parse(ctx, <<h, rem::bits>>, {_, acc, prior}) do
    parse(ctx, rem, {true, acc, <<prior::binary, h>>})
  end

  def parse(_ctx, _, _), do: {:error, "Parsing path"}

  defp maybe_add_prior(acc, <<>>), do: acc
  defp maybe_add_prior(acc, prior), do: [prior | acc]
  
end

We won't go through all the logic in this one, but suffice to say that, it will parse bytes until it reads a "/", at that point it will look to see if it has accumulated anything, in case it has it puts that as one element of the paths list. It does that until it either finds a whitespace or a ? character. If it finds a ? character it means it's expecting query parameters, so it changes its own internal acc state to now match on first extracting the value of the key for that query parameter, and once finding an = sign, switching to finding the value for that. Once it does it places those under the :query key on the context map (that is the %Satellite.Request{} struct), and it does this until again, it finds a white space.

When it finds a white space it considers this step complete. Now you know why URI's can't contain white-spaces and instead use either + or %20, imagine having to deal with that ...

Now our implementation doesn't deal with everything, it's definitively not RFC compliant, anyway, it will work mostly fine for what we want, and you can extend it by yourself, how the remaining works is left as an exercise.

Let's quit the shell and restart it, then run again:

Launchpad.start_link(%Satellite.Configuration{})

And this time visit http://localhost:4000/some/path?query1=val1

You should see now:

%Satellite.Request{accept: nil, finished_headers: false, halt: false, headers: %{}, host: nil, params: %{}, path: ["some", "path"], protocol: nil, query: %{"query1" => "val1"}, verb: :get, version: nil}

So even more success! It correctly parsed the path and separated it into each individual segment and also the query string!

Next is the Satellite.Conn, create the file at apps/satellite/lib/satellite_conn.ex and place inside it:

defmodule Satellite.Conn do

  def parse(ctx, rem, <<"HTTP/1.1", ?\r, ?\n>>), do: {:done, %{ctx | version: {1, 1}}, rem}
  def parse(ctx, rem, <<"HTTP/1.0", ?\r, ?\n>>), do: {:done, %{ctx | version: {1, 0}}, rem}

  def parse(ctx, <<>>, acc), do: {:cont, ctx, acc}

  def parse(ctx, <<?\s, rem::bits>>, acc), do: parse(ctx, rem, acc)
  def parse(ctx, <<h, rem::bits>>, acc), do: parse(ctx, rem, <<acc::binary, h>>)

  def parse(ctx, _, _), do: {:error, "Parsing Conn"}
end

Again we're not dealing with everything possible here, but for now it will work.

Set the satellite_defaults.ex to:

defmodule Satellite.Defaults do
  
  def default_pipeline() do
    [
      {:read, :verb, &Satellite.Verb.parse/3, <<>>},
      {:read, :path, &Satellite.Path.parse/3, {false, [], <<>>}},
      {:read, :conn, &Satellite.Conn.parse/3, <<>>}
    ]
  end
end

You can restart the shell, start again the launchpad and visiting localhost should now show you the version as well in the struct.

Now we'll move to the headers part, add the following line to the default_pipeline() (don't forget to add a comma on the previous last line)

{:read, :headers, &Satellite.Headers.parse/3, {:header, <<>>}}

And let's create the apps/satellite/lib/satellite_headers.ex file, and place this on it:

defmodule Satellite.Headers do

  import Satellite.Shared, only: [downcase: 1]

  def parse(ctx, <<>>, acc), do: {:cont, ctx, acc}
  def parse(ctx, <<?:, rem::bits>>, {:header, key}), do: parse(ctx, rem, {:value, key, <<>>})
  def parse(ctx, <<?\s, rem::bits>>, {:header, key}), do: parse(ctx, rem, {:header, key})
  def parse(ctx, <<?\s, rem::bits>>, {:value, key, acc}), do: parse(ctx, rem, {:value, key, acc})

  def parse(ctx, <<h, rem::bits>>, {:header, key}) do
    n_h = downcase(h)
    parse(ctx, rem, {:header, <<key::binary, n_h>>})
  end

  def parse(ctx, <<?\r, rem::bits>>, {:header, _}) do
    parse(ctx, rem, {:in_termination, <<?\r>>})
  end

  def parse(ctx, <<?\n, rem::bits>>, {:in_termination, <<?\r>>}) do
    parse(ctx, rem, {:in_termination, <<?\r,?\n>>})
  end

  def parse(ctx, <<?\r, rem::bits>>, {:in_termination, <<?\r, ?\n>>}) do
    parse(ctx, rem, {:in_termination, <<?\r,?\n,?\r>>})
  end

  def parse(ctx, <<?\n, rem::bits>>, {:in_termination, <<?\r, ?\n, ?\r>>}) do
    {:done, %{ctx | finished_headers: true}, rem}
  end

  def parse(ctx, <<h, rem::bits>>, {:in_termination, <<?\r, ?\n>>}) do
    n_h = downcase(h)
    parse(ctx, rem, {:header, <<n_h>>})
  end

  def parse(%{headers: headers} = ctx, <<?\r, rem::bits>>, {:value, key, acc}) do
    value = translate_header_content(key, acc)
    n_ctx = %{ctx | headers: Map.put(headers, key, value)}
    parse(n_ctx, rem, {:in_termination, <<?\r>>})
  end


  def parse(ctx, <<h, rem::bits>>, {:value, key, acc}) do

    n_h = case downcased_header?(key) do
            true -> downcase(h)
            _ -> h
          end
    
    parse(ctx, rem, {:value, key, <<acc::binary, n_h>>})
  end

  def parse(_ctx, _, _), do: {:error, "Parsing headers"}

  def translate_header_content(<<"content-length">>, val) when is_binary(val) do
    :erlang.binary_to_integer(val)  
    catch _ -> 0
  end

  def translate_header_content(_, val), do: val


  def downcased_header?(<<"content-type">>), do: true
  def downcased_header?(<<"accept">>), do: true
  def downcased_header?(_), do: false
end

Before doing some explanations, lets create/rewrite the apps/satellite_shared/lib/satellite_shared.ex (it should be there unless you deleted it), and place this inside of it:

defmodule Satellite.Shared do

  use Bitwise, only_operators: true
  
  Enum.each(?A..?Z, fn(value) ->
    downcased = value ^^^ 32
    def downcase(unquote(value)), do: unquote(downcased)
  end)

  def downcase(val), do: val
end

So this module uses the Bitwise macros, we specify we only want the operators type, which are in the form of ^^^ (which is the same as using the Bitwise.bxor(a, b)).

We use some simple macro magic to basically, at compile time, create a list of functions for all letters that can be uppercase that just return their downcased counterpart. We have then a catch all that just returns the arg passed into it as it is, because if it's being called the reason is the arg isn't an uppercased letter.

In erlang strings are lists of integers that are represented by their ascii code, so for instance 'A' is actually a list with one element, the integer 65, as in [65]. When erlang sees a list that is composed only of printable ascii characters it actually prints them in the shell as characters! So if you write on the shell [65, 66, 67], you'll see 'ABC'.

The elixir operator prefix ? before a character actually gives us the numeric code for that character. If you do ?A you'll see 65. So if you use a range such as ?A..?Z what will happen is that the range will be 65..90, which encompasses all the regular uppercase letters in the alphabet.

The ascii table design also has a very neat property when it comes to the alphabet letters, they're placed in such a way, that if you flip one single bit in them they become the other case, and that's what value ^^^ 32 does, so if ?A is 65, when you do that bitwise operation of bxor 32 on it it becomes 97, which is the ascii code for a so effectively you downcased the character.

Knowing this we can see that basically we define all these functions with those 5 lines of code.

def downcase(65), do: 97
def downcase(66), do: 98
... all other integers in the range
def downcase(90), do: 122
def downcase(something_else), do: something_else

And there's it! A downcase function very succintly written that doesn't do any string work, assembled at compile time. Now, this only works with ascii letters, but that's fine, because that's what we need when parsing a request at this level. We would need to add other properties for when possibly parsing url encoded characters that aren't ascii, but that is left as an exercise.

You can now also see why we used <<?some_character, ....>> in the pattern matching, basically it gives us the numeric code of the character and because binaries are at their inner structure integer sequences it works.

The headers parse function on the other hand is again following the same logic as the previous ones, it starts with an accumulator of {:header, <<>>}, and adds characters to the <<>> until it runs into a :, which signifies that the key for the header is finished, and then switches the accumulator to {:value, previously_parsed_header, <<>>} which now accumulates characters until it runs into \r which will be the ending sequence, so we change the acc to be :in_termination mode, when followed by \n indicates the header and its value has finished, if it's followed by another \r\n sequence then it means all headers have been parsed, if it's not we know we're in another header key and we change to start accumulating the new header key and then value and then entering again the termination sequence until we hit \r\n\r\n, and we're now done with the request unless it has a body too, which it won't in a get request.

The headers keys should be case insensitive, so we always downcase them. We also add some logic for certain headers that we want their values to also be always downcased, namely content-type and accept. We could add additional ones or remove these if we wanted to. We also add a special case for the content-length header in case it's present, before we set it in the request struct, we try to translate it into an integer value because that will be more useful if we wanted then to use the value to count how many bytes we need to parse or whatever.

If you restart the iex console and the launchpad, and then visit localhost you should see that now we also have the headers key filled with all headers neatly mapped.

The same way we downcase the value for certain headers, we could also create, if wanted, a special white-list of header keys to be placed as :atoms in our headers map, but that would be confusing by mixing both atom and string keys so we won't worry with that now.

Let's add this new pipeline element to apps/satellite/lib/satellite_defaults.ex

{:noread, :host, &Satellite.Host.set/2, nil}

And create the respective module in apps/satellite/lib/satellite_host.ex with the following content:

defmodule Satellite.Host do

  import Satellite.Shared, only: [downcase: 1]

  def set(%{headers: headers} = ctx, _) do
    host = Map.get(headers, "host", "")
    {:ok, %{ctx | host: split_host(host)}}
  end

  def split_host(val), do: split_host(val, <<>>, [])
  def split_host(<<>>, segment, acc), do: Enum.reverse([segment | acc])
  def split_host(<<?., rem::bits>>, segment, acc), do: split_host(rem, <<>>, [segment | acc])
  def split_host(<<?:, _::bits>>, segment, acc), do: Enum.reverse([segment | acc])
  def split_host(<<h, rem::bits>>, segment, acc) do
    n_h = downcase(h)
    split_host(rem, <<segment::binary, n_h>>, acc)
  end

  def split_host(_, _, _), do: []
end

This step in the pipeline is new, so we also have to implement our handle_event for this particular case.

Place this handle_event function between the existing ...:internal, :wait and ...:internal, :read, the order in which these appear is important, because otherwise, if this was after the :internal, :read that would be the one being matched and it would hang and timeout waiting for new data on the socket, since it doesn't care about which state we're in. By placing this one before, we actually make it so that when the state is of type {:noread, ....} it will be called instead of anything else.

def handle_event(:internal, event, {:noread, _name, fun, acc}, %{request: request} = data) do
    case fun.(request, acc) do
      {:ok, n_request} ->
        set_next_step(%{data | request: n_request}, {:event, event})

      {:error, reason} ->
        {:next_state, :response, %{data | request: "Error: #{inspect reason}"}, [{:next_event, :internal, :send_response}]}
    end
end

Now, the event that is matched as the second argument already contains either {:parse, data} or :read as its event, so we will pass the remaining to set_next_step as a new type of tuple, {:event, event} and change our set_next_step function to account for this new type by changing it to:

defp set_next_step(%{pipeline: [h | t]} = data, remaining) do

    event = case remaining do
              <<>> -> :read
              {:event, event} -> event
              _ -> {:parse, remaining}
            end

    {:next_state, h, %{data | pipeline: t}, [{:next_event, :internal, event}]}
end

This way, it sets the already existing defined event as the next one. Let's restart iex and launchpad and make sure its still working, we should now see the host key populated as well. Again, we're setting it as a list, instead of a plain string, because, imagine we would want to distinguish between requests in our router that went to a certain subdomain, by having it as a list split by the periods, we will be able to easily encode that logic in our dispatcher/router, which is not evident when you only see a list with a single element ["localhost"], but if it was ["mysubdomain", "domain", "com"] things change, the same if we wanted to have localised logic pertaining to the top domain, then it becomes more evident why it's a good idea.

Next on the pipeline will be this:

{:noread, :accept, &Satellite.Accept.set/2, nil}

Since we already wrote the handle for the :noread case we just need to write a module at apps/satellite/lib/satellite_accept.ex with:

defmodule Satellite.Accept do

  def extract(%{headers: headers} = ctx, _) do
    {:ok, %{ctx | accept: extract_accept(headers)}}
  end

  def extract_accept(%{"accept" => <<"application/json", _::bits>>}), do: :json
  def extract_accept(%{"accept" => <<"text/html", _::bits>>}), do: :html
  def extract_accept(_), do: :any
    
end

Now this is very basic, perhaps we would like to do as we did with paths and hosts to split all accepted content-types and create a list of that that we could further match down the road but for now this will do, if the first characters of the header are application/json or text/html we set it to :json or :html respectively, anything else we set to :any (which is obviously incorrect, but we would also need to set up mimes and... we'll leave it for a future occasion).

Restart iex and the launchpad, you should now see the accept key also populated.

We're getting closer to finishing our first version of our server. Add this new step to the pipeline:

{:check, :check_request_type, &Satellite.Check.check/2, nil}

And create apps/satellite/lib/satellite_check.ex with:

defmodule Satellite.Check do

  def check(%{verb: :get} = ctx, _), do: {:dispatch, ctx}
  def check(%{verb: :post} = ctx, _), do: {:next, ctx}
  def check(ctx, _), do: {:dispatch, ctx}
end

This of course is not enough to cover everything an http request might have, but is enough for our sample, if it's a get request we'll immediately run the not yet written dispatch event and if it's a post we will actually want to continue parsing. Now we need to write the corresponding handling for this event with those possible return values on our satellite.ex gen_statem:

def handle_event(:internal, event, {:check, _name, fun, acc}, %{request: request} = data) do
    case fun.(request, acc) do
      {:next, n_request} ->
        set_next_step(%{data | request: n_request}, {:event, event})

      {:dispatch, n_request} ->
        {:next_state, :dispatching, %{data | request: n_request}, [{:next_event, :internal, :dispatch}]}

      {:response, resp} ->
        {:next_state, :response, %{data | request: resp}, [{:next_event, :internal, :send_response}]}

      {:error, reason} ->
        {:next_state, :response, %{data | request: "Error, #{inspect reason}"}, [{:next_event, :internal, :send_response}]}
    end
end

Again this needs to be placed before the handle_event(:internal, :read.....) function. There's some things we haven't implemented yet, like the :dispatch event and in the {:error, ...} or {:response, ...} we're just doing a very silly thing, so to correct that we will now work on the router and afterwards implement the request dispatcher and lastly the body parsing step. If we restart the shell and launchpad we'll now hit an error when visiting the page because the next event :dispatch isn't implemented yet.

For the router we'll use some of elixir's macro magic. Macros have some complexity but also allow us to write some really useful code. We'll try to not abuse their usage and basically write code that is still readable, with the main difference being it's written to be run at compile time. I'm no expert in elixir macros, so I won't explain everything (there's much I don't understand also), but basically they allow you to define AST code to be used in code generation at compile time.

What we want to achieve is the ability to create a module and then be able to write:

route "get", "/some/:path", AnotherModule, :a_function

(just like Phoenix does for us, but our version will be very plain and lacking many other features)

And with this create a route that accepts a get request, on a path composed by /some/*anything* and calls a :a_function on AnotherModule passing it the parsed request, and in the process setting a variable in the params with key :path set to the whatever was the second segment of that path.

So we want to accept literal paths, path segments to be bound to a named key (prefixed by :), or any segment (*). We also want for each route to be able to accept optionally a host/domain to be matched, defaulting to * (any), and again offering the possibility of binding those parts of the domain to named keys in the params map that will be populated in the request. Create the file apps/satellite/lib/satellite_routing.ex and place in it:

defmodule Satellite.Routing do


  defmacro __using__(_opts) do
    quote do
      import Satellite.Routing
      @before_compile Satellite.Routing
    end
  end

  defmacro route(verb, path, module, function, domain \\ "*") do

    verb_atom = String.to_atom(String.downcase(verb))
    {path_splitted, vars} = split_path(path)
    {domain_splitted, domain_vars} = split_domain(domain)
    all_vars = vars ++ domain_vars
    
    quote do
      def route(unquote(verb_atom), unquote(path_splitted), unquote(domain_splitted), %{params: params} = request) do
        ctx = %{
          request |
          params: Enum.reduce(unquote(all_vars), params, fn({key, var}, acc) ->
            Map.put(acc, key, Macro.escape(var))
          end)
        }
        
        apply(unquote(module), unquote(function), [ctx])
      end
    end
  end

  defp split_path(path) do
    case String.split(path, "/", trim: true) do
      ["*"] -> {(quote do: _), []}
      
      split ->
          Enum.reduce(split, {[], []}, fn
            ("*", {acc1, acc2}) ->
              {[(quote do: _) | acc1], acc2}
            
            (<<?:, rem::binary>>, {acc1, acc2}) ->
              {[Macro.var(String.to_atom(rem), nil) | acc1], [{String.to_atom(rem), Macro.var(String.to_atom(rem), nil)} | acc2]}
            
            (other, {acc1, acc2}) ->
              {[other | acc1], acc2}
            
          end)
          |> case do
               {paths, vars} -> {Enum.reverse(paths), Enum.reverse(vars)}
             end
    end
 end

  defp split_domain(domain) do
    case String.split(domain, ".", trim: true) do
      ["*"] -> {(quote do: _), []}
      
      [<<?:, rem::binary>>] ->
          host_var = String.to_atom("host_#{rem}")
          {Macro.var(host_var, nil), [{host_var, Macro.var(host_var, nil)}]}

        split ->
        Enum.reduce(split, {[], []}, fn
          ("*", {acc1, acc2}) ->
            {[(quote do: _) | acc1], acc2}

          (<<?\\, rem::binary>>, {acc1, acc2}) -> {[rem | acc1], acc2}
          
          (<<?:, rem::binary>>, {acc1, acc2}) ->
            host_var = String.to_atom("host_#{rem}")
            {[Macro.var(host_var, nil) | acc1], [{host_var, Macro.var(host_var, nil)} | acc2]}
          
          (other, {acc1, acc2}) ->
            {[other | acc1], acc2}
          
        end)
        |> case do
             {paths, vars} -> {Enum.reverse(paths), Enum.reverse(vars)}
           end
    end
  end


  defmacro __before_compile__(_env) do
    quote do
      def route(_, _, _, _ctx) do
        Satellite.Response.not_found()
      end
    end
  end
end

We start by defining a __using__ macro. This is a special macro definition that then allows us to write use Name_of_the_module_where_the_using_is_defined and it automatically makes the contents of block available in that module.

This __using__ does only two things, it imports the module to where it is invoked and sets a hook to be run before that module that calls use is compiled.

Then we define our actual route macro. This macro accepts 4 or 5 arguments, defaulting the last one to "*" (the domain). It takes the verb to match, the path, the module to call, and the function in that module to call and, optionally a domain string to match.

With this info it makes some transformations, the verb to an atom (to match what our parser creates), splits the path into its own individual segments - it does this by a regular function, and with help of Macro.var creates representations for variables -

this is what allows us to actually transform a segment such as :something, into a variable in the function definition, and then use that and the parsed named to set a key in the params map with the value that that variable will be bound to when ran at runtime. Since this is creating a AST representation of code it doesn't have any values right now, it just has a "representation" of what that code will be when run -

and does the same for the domain. On the splitting functions, you can see that we treat the "*" match in the domain and path in a special way when it's the only segment. Because if we treated it as other segments, the generated code would look like [ _ ]. If then someone did a request and our host was host.domain.com, this would be split into ["host", "domain", "com"], and that wouldn't match a list with a single value like [ _ ], so we need to treat it specially so that the whole thing is simply _, this way it matches anything like the wildcard actually means, either ["localhost"], or ["some", "domain", "com"].

Then, because we do quote do and inside it define a function, when the macro is called from some other module, what will actually happen is that this function definition we're creating is written in that module with the arguments we prepared.

So let's say again we have this module:

defmodule Test.Router do
    use Satellite.Routing

    route "get", "/some/:path", AnotherModule, :a_function
end

(the route macro could be written as route("get", "/some/:path", AnotherModule, :a_function), but we can omit parenthesis and in this case it makes it look more declarative, like the Phoenix Router).

When elixir compiles this particular module with that sample code, what actually ends up being defined is

defmodule Test.Router do
    def route(:get, ["some", path], _, %{params: params} = request) do
        ctx = %{
            request | 
            params:  
                Enum.reduce([{:path, ast_representation_of_the_var_variable}], params, fn({key, var}, acc) ->
                    Map.put(acc, key, var)
                end)
            }

        apply(AnotherModule, :a_function, [ctx])
	end

    def route(_, _, _, _ctx), do: Satellite.Response.not_found()
end

The last route definition comes from the @before_compile Satellite.Routing declaration in the __using__ macro, which basically allows us to run that macro function that defines that route, after the other macros have been run, but before compilation, so they end up having the correct order.

We can see how we can use this in our satellite gen_statem to dispatch the requests since we have the verb, the path and domain neatly decomposed into segments from the parsing we do in the pipeline.

So now we need to change our satellite.ex file. The changes are mostly self-evident but to prevent errors I'm just going to past the full contents of the file as it should be:

defmodule Satellite do
  @behaviour :gen_statem

  alias Satellite.{Request, Response}
  
  defstruct [:socket, :config, :conn, :response, :router, pipeline: [], request: %Request{}]
  
  @impl true
  def callback_mode(), do: :handle_event_function
  
  def start_link(socket, opts) do
    :gen_statem.start_link(__MODULE__, {socket, opts}, [])
  end
  
  @impl true
  def init({socket, config}) do
    {:ok, :waiting,  %__MODULE__{socket: socket, config: config}, [{:next_event, :internal, :wait}]}
  end

  @impl true
  def handle_event(:internal, :wait, :waiting, %{socket: socket, config: %{router: router, pipeline: pipeline}} = data) do
    {:ok, conn} = :gen_tcp.accept(socket)
    
    :gen_tcp.controlling_process(conn, self())

    n_data = %{data | conn: conn, router: router, pipeline: pipeline, request: %Request{}}

    set_next_step(n_data, <<>>)
  end

  def handle_event(:internal, event, {:noread, _name, fun, acc}, %{request: request} = data) do
    case fun.(request, acc) do
      {:ok, n_request} ->
        set_next_step(%{data | request: n_request}, {:event, event})

      {:error, reason} ->
        response = Response.error_resp("#{inspect reason}")
        {:next_state, :response, %{data | response: response}, [{:next_event, :internal, :send_response}]}
    end
  end

  def handle_event(:internal, event, {:check, _name, fun, acc}, %{request: request} = data) do
    case fun.(request, acc) do
      {:next, n_request} ->
        set_next_step(%{data | request: n_request}, {:event, event})

      {:dispatch, n_request} ->
        {:next_state, :dispatching, %{data | request: n_request}, [{:next_event, :internal, :dispatch}]}

      {:response, response} ->
        {:next_state, :response, %{data | response: response}, [{:next_event, :internal, :send_response}]}

      {:error, reason} ->
        response = Response.error_resp("#{inspect reason}")
        {:next_state, :response, %{data | response: response}, [{:next_event, :internal, :send_response}]}
    end
  end

  def handle_event(:internal, :read, _, %{conn: conn} = data) do
    case :gen_tcp.recv(conn, 0, 1000) do
      {:ok, packet} ->
        IO.inspect(packet, label: "packet")
        {:keep_state_and_data, [{:next_event, :internal, {:parse, packet}}]}

      {:error, reason} ->
        response = Response.error_resp("#{inspect reason}")
        {:next_state, :response, %{data | response: response}, [{:next_event, :internal, :send_response}]}
    end
  end

  def handle_event(:internal, {:parse, packet}, {type, name, fun, acc}, %{request: request} = data) do

    case fun.(request, packet, acc) do
      {:cont, n_request, n_acc} ->
        IO.inspect({:cont, n_request, n_acc})
        {:next_state, {type, name, fun, n_acc}, %{data | request: n_request}, [{:next_event, :internal, :read}]}

      {:done, n_request, remaining} ->
        IO.inspect({:done, n_request, remaining})
        set_next_step(%{data | request: n_request}, remaining)

      {:error, reason} ->
        response = Response.error_resp("#{inspect reason}")
        {:next_state, :response, %{data | response: response}, [{:next_event, :internal, :send_response}]}
    end
    
  end

  def handle_event(:internal, :dispatch, :dispatching, %{request: request, conn: conn, router: router} = data) do
    try_dispatch(conn, router, request)
    {:next_state, :waiting, %{data | conn: nil}, [{:next_event, :internal, :wait}]} 
  end

  def handle_event(:internal, :send_response, :response, %{conn: conn, response: response} = data) do
    try_send_response(conn, response)
    {:next_state, :waiting, %{data | conn: nil}, [{:next_event, :internal, :wait}]}
  end

  def handle_event(:internal, :close, _, %{conn: conn} = data) do
    :gen_tcp.close(conn)
    {:next_state, :waiting, %{data | conn: nil}, [{:next_event, :internal, :wait}]}
  end


  defp set_next_step(%{pipeline: [h | t]} = data, remaining) do

    event = case remaining do
              <<>> -> :read
              {:event, event} -> event
              _ -> {:parse, remaining}
            end

    {:next_state, h, %{data | pipeline: t}, [{:next_event, :internal, event}]}
  end

  defp set_next_step(%{pipeline: []} = data, _remaining) do
    {:next_state, :dispatching, data, [{:next_event, :internal, :dispatch}]}
  end

  defp try_dispatch(conn, router, %{verb: verb, path: path, host: host} = request) do
    case apply(router, :route, [verb, path, host, request]) do
      response -> try_send_response(conn, response)
    end
  rescue
    e -> try_send_response(conn, Response.error_resp())
  end

  defp try_send_response(conn, %Response{} = response) do
    n_response = Response.make_resp(response)
    send_response(conn, n_response)
  after
    :gen_tcp.close(conn)
  end

  defp try_send_response(conn, response) when is_binary(response) do
    send_response(conn, response)
  after
    :gen_tcp.close(conn)
  end

  defp send_response(conn, response) do
    :gen_tcp.send(conn, response)
  end
end

And we need to define also, a module to handle responses, create a file apps/satellite/lib/satellite_response.ex with:

defmodule Satellite.Response do

  defstruct [code: 200, headers: [{<<"content-type">>, <<"text/html">>}], body: <<>>]

  @codes [{200, "OK"}, {404, "Not Found"}, {500, "Internal Server Error"}]
  
  def make_resp(
    %__MODULE__{
      code: code,
      headers: headers,
      body: body
    }
  ) do

    code_prep = make_code(code)
    headers_prep = map_headers(headers)
    {n_body, content_length_prep} = create_length(body)
    
    <<"HTTP/1.0 ", code_prep::binary, "\n", headers_prep::binary, content_length_prep::binary, "\n", n_body::binary>>
  end

  defp map_headers(headers), do: map_headers(headers, <<>>)
  defp map_headers([{header, value} | t], acc), do: map_headers(t, <<acc::binary, header::binary, ": ", value::binary, "\n">>)
  defp map_headers([], acc), do: acc
  
  defp create_length(nil), do: {<<>>, <<"content-length: 0">>}
  defp create_length(<<>>), do: {<<>>, <<"content-length: 0">>}
  defp create_length(body) do
    size = :erlang.integer_to_binary(:erlang.size(body))
    {body, <<"content-length: ", size::binary, "\n">>}
  end


  Enum.each(@codes, fn({code, val}) ->

    string_v = Integer.to_string(code)
    atom_v = String.to_atom(string_v)

    defp make_code(unquote(code)), do: <<unquote(string_v)::binary, " ", unquote(val)::binary>>
    defp make_code(unquote(string_v)), do: <<unquote(string_v)::binary, " ", unquote(val)::binary>>
    defp make_code(unquote(atom_v)), do: <<unquote(string_v)::binary, " ", unquote(val)::binary>>

  end)

  def error_resp(body \\ "Internal Server Error") do
    %__MODULE__{code: 500, headers: [{<<"content-type">>, <<"text/html">>}], body: body}
  end

  def not_found(body \\ "Not Found") do
    %__MODULE__{code: 404, headers: [{<<"content-type">>, <<"text/html">>}], body: body}
  end
  
end

And lastly, just to test that everything goes according to plan, lets create a test router and a controller for it. apps/satellite/lib/test_router.ex:

defmodule Test.Router do
  use Satellite.Routing

  route "get", "/", Test.Controller, :test
  route "get", "/:any/oi/:some", Test.Controller, :test2, "*"
  route "get", "*", Test.Controller, :test3
  route "post", "/data", Test.Controller, :test4
end

defmodule Test.Controller do

  def test(request) do
    %Satellite.Response{body: "this is the root path, nothing to see here"}
    |> Satellite.Response.make_resp()
  end

  def test2(request) do
    %Satellite.Response{body: "#{inspect request}"}
    |> Satellite.Response.make_resp()
  end

  def test3(request) do
     %Satellite.Response{body: """
	<html><body><h1>Wildcard match!</h1><br><br><div style="color: red;">#{inspect request}</div></body></html>
      """}
      |> Satellite.Response.make_resp()
   end
  
  def test4(%{body: parsed_body} = request) do
     response_body =  "Parsed: " <> Jason.encode!(parsed_body) <> "\n"
     %Satellite.Response{body: response_body, headers: [{<<"content-type">>, <<"application/json">>}]}
     |> Satellite.Response.make_resp()
   end

end

Since we'll also want to test post requests and we'll use Jason to encode JSON, we might as well go ahead and add the last bit to our pipeline, the body parser. Right now we'll only do it for JSON.

On apps/satellite/lib/satellite_defaults.ex add the following {:read, :body, &Satellite.Body.parse/3, {0, <<>>}}

The full pipeline should now look like:

defmodule Satellite.Defaults do
  
  def default_pipeline() do
    [
      {:read, :verb, &Satellite.Verb.parse/3, <<>>},
      {:read, :path, &Satellite.Path.parse/3, {false, [], <<>>}},
      {:read, :conn, &Satellite.Conn.parse/3, <<>>},
      {:read, :headers, &Satellite.Headers.parse/3, {:header, <<>>}},
      {:noread, :host, &Satellite.Host.set/2, nil},
      {:noread, :accept, &Satellite.Accept.set/2, nil},
      {:check, :check_request_type, &Satellite.Check.check/2, nil},
      {:read, :body, &Satellite.Body.parse/3, {0, <<>>}}
    ]
  end
  
end

(note: we kept it as tuples, because tuples are the fastest way to access something, if in erlang we would use Records which are still tuples underneath and in Elixir we would use proper structs to encode this, something like %Satellite.Step{mode: :read, name: :verb, fun: &Satellite.Verb.parse/3, acc: <<>>}, this would be much more structured, specially with @enforce_keys we could give some guarantees on when placing a step in the pipeline but that's all out of scope for this tutorial)

And create the file apps/satellite/lib/satellite_body.ex:

defmodule Satellite.Body do

  def parse(%{verb: :post, headers: headers} = request, rem, {count, acc}) do
    n_size = count + :erlang.size(rem)
    case headers do
      %{<<"content-length">> => 0} -> {:done, request, <<>>}
      %{<<"content-length">> => ^n_size} ->
        case parse_content(headers, <<acc::binary, rem::binary>>) do
          {:ok, decoded} -> {:done, %{request | body: decoded}, <<>>}
          {:error, error} -> {:error, error}
        end
      _ ->
        {:cont, request, {n_size, <<acc::bits, rem::bits>>}}
    end
  end

  def parse(request, _, _), do: {:done, request, <<>>}

  def parse_content(%{<<"content-type">> => <<"application/json">>}, rem) do
    Jason.decode(rem)
    catch e -> {:error, e}
  end
  
end

Lastly, we need to add Jason as a dependency so we can use it, on apps/satellite/mix.exs change your deps function to this:

defp deps do
    [
      {:satellite_shared, in_umbrella: true},
      {:jason, "~> 1.0"}
    ]
end

After that, save the file, quit the shell and do:

mix deps.get

This will install the dependencies.

With this in place, let's exit the shell, restart it with iex -S mix run and then do Launchpad.start_link(%Satellite.Configuration{router: Test.Router})

Now visit http://localhost:4000, http://localhost:4000/something/oi/else and http://localhost:4000/anything_goes

You should see the correct ouput in the browser.

Now to test posts we'll use curl with the -v flag (verbose) option to see if it works. Open a new terminal window (but don't close the one running the server ofc), and execute:

curl -v -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST http://localhost:4000/data

You should see an output of the request details, finished by:

* upload completely sent off: 34 out of 34 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< content-type: application/json
< content-length: 42
< 
Parsed: {"key1":"value1","key2":"value2"}
Closing connection 0

If you try:

curl -v -d 'oiioi' -H "Content-Type: application/json" -X POST http://localhost:4000/data

You should see instead:

%Jason.DecodeError{data: "oiioi", position: 0, token: nil} as the response, which is the error from trying to parse invalid json "oiioi". Of course this is also not how it should in reality be implemented, it should not leak the details of what is wrong internally, but we wrote it in such a way to illustrate that although the request raised an error, it was still answered.

Now one last thing is to see if we can use this on another app. So let's remove the file apps/satellite/lib/test_router.ex, go outside the folder of the umbrella and create a new app, with mix new test_satellitex --sup

Then add to test_satellitex/lib two files, router.ex and controller.ex, on router.ex add:

defmodule Router do
  use Satellite.Routing

  route "get", "/", Controller, :test
  route "get", "/:any/oi/:some", Controller, :test2, "*"
  route "get", "*", Controller, :test3
  route "post", "/data", Controller, :test4
end

And on controller.ex add:

defmodule Controller do

  def test(request) do
    %Satellite.Response{body: "#{inspect request}"}
    |> Satellite.Response.make_resp()
  end

  def test2(request) do
    %Satellite.Response{body: "#{inspect request}"}
    |> Satellite.Response.make_resp()
  end

  def test3(request) do
    %Satellite.Response{body: """
    <html><body><h1>Wildcard match!</h1><br><br><div style="color: red;">#{inspect request}</div></body></html>
    """}
    |> Satellite.Response.make_resp()
  end

  def test4(%{body: parsed_body}) do
    response_body = "Parsed: " <> Jason.encode!(parsed_body) <> "\n"
    %Satellite.Response{body: response_body, headers: [{<<"content-type">>, <<"application/json">>}]}
    |> Satellite.Response.make_resp()
  end

end

On test_satellitex/mix.exs change the deps to:

defp deps do
    [
      {:satellitex, path: "../satellitex"}
    ]
end

(assuming you created the test_satellitex app on the same folder level as satellitex, if not change the relative path accordingly). We're using a local path just to test, in a real situation this would be either in git or hex and you would use either a git path or a normal dep definition.

Save the file, and edit test_satellitex/lib/test_satellitex/application.ex to:

defmodule TestSatellitex.Application do
  @moduledoc false

  use Application

  def start(_type, _args) do
    children = [
      Supervisor.child_spec(%{id: Server1, start: {Launchpad, :start_link, [%Satellite.Configuration{router: Router}]}}, type: :worker)
    ]

    opts = [strategy: :one_for_one, name: TestSatellitex.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Then from the root dir of this app do mix deps.get to install the dependencies. And then start it with:

iex -S mix run

And you should have a running server on another application.

And that's it! This post is already gigantic, so we'll stop here. It's very, very far from being a compliant http server, there's many things missing but the general structure is laid out, about what is missing:

• parsing cookies!
• serving static assets
• appropriate response handlers for different formats other than html/json
• mime support
• html full range of response codes
• http protocol version negotiation
• ssl/tls
• websockets
• another type of callback in our gen_statem besides :read, :noread and :check, probably named :connection where the called function receives the request and the socket itself, so if wanted it could retrieve information from the socket such as the peer_address (ip address of whoever is making the request, amongst other things) - this would allow to easily create a rate-limiter as the first step of the pipeline, if wanted without actually reading anything from the socket
• preparing templates & "views" to send as responses
• defining the pipeline elements as behaviours so that it would be easier to understand what you need to do to have a "compliant" pipeline step created

And many of those are really grunt work, extensive research (reading the RFC's and making the implementation be spec compliant) and/or tricky to implement correctly, so hopefully, besides having learned some things about gen_statems, erlang and elixir it also increased your appreciation for the existing libraries that do all of this, correctly and efficiently and available for free!

]]> https://micaelnussbaumer.com/posts/pattern-matching-in-elixir Sat, 11 May 2019 01:00:00 UTC https://micaelnussbaumer.com/posts/pattern-matching-in-elixir Pattern matching is a very powerful feature in any programming language that implements it, I think that essentially because pattern matching is inherent in computer programming logic, from a conceptual to a practical level. It's a feature that is present in many functional languages for the programmer to use. Elixir (inheriting from Erlang) gives us access to it and idiomatic code is most often peppered with pattern matching.

It fits well in many places and allows us to describe very succinctly the code paths our running programs may take, while retaining, and most of the time increasing, the clarity of the code. While some features that allow succinctness and conciseness of code end up in a symbolical soup more close to mathematics than a "legible" declaration (for those of us not completely familiar with mathematical notation), pattern matching does not suffer from that, because what it allows us to express is the "form" of the data inside our running programs, which in turn allows us to express what pathways of code should be run given that form.

Erlang itself has on top of that a very accessible and intuitive way of treating binary data, that coupled with pattern matching gives it the ability to chop, split, and work with binaries in a very pleasant way.

Now I could go on talking about how the rainbow is perfect during this sunset, or how unicorns are pooping diamonds and so on, but that won't give you any insight into Pattern Matching. So let's move to do some exploration of it.

First we'll start by analysing the = operator in Elixir, which is in fact a match operator and not an assignment operator (if you're coming from Javascript, Ruby, etc). It has one side effect that makes it look like an assignment operator though, which is, when the expression on the left side of it is a variable, it binds whatever is on the right side to that variable.

We can do some simple tests of it on iex:

1 = 1
# 1
1 = 2
# ** (MatchError) no match of right hand side value: 2

a = 1
# 1

a = 2
# 2

a + 1
# 3

a
# 2

3 = a
#** (MatchError) no match of right hand side value: 2

a = 3
# 3

a = a + 1
# 4

a
# 4

Ok, so what can we see from these examples? First we see that when a match evaluates correctly, we get the value of the expression that was matched. 1 = 1 returns 1, and not true. If we wanted to check for equality we would use the equality operator == and not the match operator =.

> 1 == 1
# true

> 1 == 2
# false

On the second case we get a MatchError, because we're trying to convince our program that 1 matches 2, and it simply states, dude, perhaps in your universe, but in mine, 1 can't match 2.

Then we do a bunch of "assignments", and it works mostly as one would expect in other languages, the variable a gets bound to whatever value.

When we try to do 3 = a, when a was bound to 2, we get again an error, but when we switch it to a = 3 we no longer get an error, instead the value 3 is bound to the variable a.

In Erlang you can't use the same variable name in different assignments, but Elixir chose to allow it. Underneath, because the data is still immutable, what happens is that elixir creates various versions of a and points a to the last version you entered - from a practical point of view though, it looks like you're "re-assigning" the variable.

Now you might be thinking, all of that for what? Yes, it's not apparent yet why it helps in anything else other than silly examples. So let's move on.

Let's create a map and play around with it.

a = %{key_one: "this is key number one", key_two: "this is key number two"}
# %{key_one: "this is key number one", key_two: "this is key number two"}

%{key_one: key_one_var} = a
# %{key_one: "this is key number one", key_two: "this is key number two"}

key_one_var
# "this is key number one"

So here things start to look more interesting. You can see that we placed a pattern of %{key_one: key_one_var} and matched it against the previously map bound to a. The match succeeded, but we also "bound" the variable key_one_var to whatever was the value of the key :key_one used in the pattern match. If you used Javascript E6 destructuring you might notice it looks familiar. The Erlang (&Elixir) version though is way more powerful than javascript's version, because of its properties and how it can be used, but lets move on.

%{key_three: key_three_var} = a
# ** (MatchError) no match of right hand side value: %{key_one: "this is key number one", key_two: "this is key number two"}

Ok, we tried to match the map bound to a to a map of the form %{key_three: some_variable} and this didn't work, because a doesn't have any key named :key_three, it's expected to fail. But notice that we didn't need to specify all the keys when matching :key_one previously, although the a map had an additional :key_two.

So how can we look at it conceptually? I think that the best way to describe it is, given a = b, could the form described in a be extracted from the contents held in b? If it can the match succeeds, any bindings are made effective, and the full expression is returned. If it can't, an error is thrown.

And in this case indeed it can, in the previous example, the form map, with a key named :key_one, to be bound to a variable named key_one_var can be extracted from a, because that map has a key named :key_one and we aren't specifying any value that that key must have.

So if we write:

%{key_one: "this is key number one"} = a

It also matches. But if we write:

%{key_one: "this is key number two"} = a
** (MatchError) no match of right hand side value: %{key_one: "this is key number one", key_two: "this is key number two"}

It fails, because we're saying match a map that has a key named :key_one with the value "this is key number two", but since the value of that key in a is actually "this is key number one" it fails.

%{key_one: "this is key number " <> keynumber} = a
# %{key_one: "this is key number one", key_two: "this is key number two"}

keynumber
# "one"

Here it matches, and not only that, we have extracted the last portion of the binary string as a variable. We could extract both the last portion and the whole binary string if we wanted with for instance:

%{key_one: "this is key number " <> keynumber = whole_binary} = a
# %{key_one: "this is key number one", key_two: "this is key number two"}

keynumber
# "one"

whole_binary
# "this is key number one"

It doesn't end here though, because the patterns you use can be much more complex and involve nested levels of maps, binary specifications, and lists, and so on, you can "describe" in as much detail as you want their forms and also extract very easily what you need from them.

%{} = a

Matches alright again, because we're simply saying, is the value bound to a a map? And since it is, all is good. You might have thought that it would fail, because %{} could be interpreted as an empty map but maps behave in this regard a bit differently. To assert that a map is indeed empty, you need to compare it, %{} == a would return false for instance.

Now lists:

b = ["a", "list", :of, :stuff]
# ["a", "list", :of, :stuff]

[head | tail] = b
# ["a", "list", :of, :stuff]

head
# "a"

tail
# ["list", :of, :stuff]

Here we bound b to a list of 4 elements. Then we pattern-matched b against the pattern [head | tail]. Lists in Erlang (and consequently in Elixir) are like LISP lists, they're composed of cons cells, that can be thought of as a group of cells, where each cell holds a value and also a "pointer" to the next cell in the list.

In this case we can think of it as a structure where the first "cell" is "a", which points to the cell with value "list", which in turn points to the cell :of, which points to cell :stuff which in turn points to the end of the list (an empty list, []).

So a list is a collection of cells where each element holds its value and points to the next element, that's why they're called cons cells and usually described as (x . points_to_y)(y . points_to_z)(z . points_to_empty_list)(), or more correctly (x . (y . (z . ()))). In Elixir and Erlang it looks like [x | [y | [z | []]]].

So when we match b to the pattern [head | tail] what we're actually saying is, does a pattern of a non empty list (meaning where there is at least one cell, the head) match b? If it does, bind the value of the first cell of that list to the variable head and the remaining of the list to the variable tail.

If we try:

c = []
# []

[head | tail] = c
# ** (MatchError) no match of right hand side value: []

We get a match error, in this case we're trying to match a non-empty list (the pattern we wrote specifies it should have one element at least, head, pointing to a tail), against an empty list (c) so we get a match error, because an empty list has no cells, it's itself the end of the list. This contrasts with the previous seen case of the map, where the empty map %{} still matched ok, but in practical (and theoretical) terms it makes sense, once you start using it you'll see that the difference between what meaning an empty list and an empty map usually assume and the ability to match on simply being a map warrant this (seemingly) small contradiction. Also, if a match with %{} equated to saying an empty map, then when you were matching non-empty maps you would need to spell out all the keys in the map and that effectively and that completely defeats the pragmatic purpose of matches - or you would need to specify different behaviour for when using the pattern %{}.

d = ["non_empty"]
# ["non_empty"]

[head | tail] = d
# ["non_empty"]

head
# "non_empty"

tail
# []

In this case though, we bound a list with one cell, "non_empty" to d and when we pattern matched, it worked. "non_empty" got bound to head and the end of the list (an empty list) got bound to tail.

Again, it's not super impressive (yet) but we'll get there soon. Let's see one more example before though:

deep = %{list_key: [:a_list, %{super_deep: [:a]}], date: "2019-05-01"}
# %{date: "2019-05-01", list_key: [:a_list, %{super_deep: [:a]}]}

%{date: <<year::binary-size(4), "-", month::binary-size(2), "-", day::binary-size(2)>>, list_key: [_, %{super_deep: [first_element_of_super_deep | t]}]}
# %{date: "2019-05-01", list_key: [:a_list, %{super_deep: [:a]}]}

> year
# "2019"
> month
# "05"
> day
# "01"
> first_element_of_super_deep
# :a

Wow. So with a simple pattern matching, we were able to extract a lot of information as you can see. We didn't need to split a string to get all the pieces of the date, we didn't need to iterate on the list to get it's nested elements and we plucked an element from inside a list, that was inside a map, that was inside another list itself inside another map. We can also write it in a way that is more readable

%{
    date: <<
        year::binary-size(4),
        "-",
        month::binary-size(2),
        "-",
        day::binary-size(2)
    >>,
    list_key: [
        _,
        %{
            this_will_fail: [
                first_element_of_super_deep | t
            ]
        }
    ]
} = deep

# ** (MatchError) no match of right hand side value: %{date: "2019-05-01", list_key: [:a_list, %{super_deep: [:a]}]}

Here, in the pattern we described, we changed the key name inside the map inside the nested list to this_will_fail and it no longer matched, although everything else was the same as before.

So, this is a bit more impressive, although, still doesn't look very useful if we can only use this on match operations. Where it becomes really, really, useful is when we use it in conjunction with Elixir's (Erlang) ability to have multiple function definitions and/or inside some constructs the language provides, such as case statements (Well actually, most places, to be sincere, allow their usage).

You might also have noticed that we used _ in these last pattern matches. _ (or any variable name starting with _) tells the compiler we're not interested in that value, so it won't bind that variable, although it still requires something to be there.

So let's look at patterns with case statements, since they're very common too.

a = [:a, :list]
# [:a, :list]

case a do
    [:a, :list] -> "a list with :a and :list"
    _ -> "something else"
end
# "a list with :a and :list"

a = [:b, :list]
# [:b, :list]

case a do
    [:a, :list] -> "a list with :a and :list"
    _ -> "something else"
end
# "something else"

case a do
    [_, :list] -> "a two element list where the second element is :list"
    _ -> "something else"
end
# "a two element list where the second element is :list"

case a do
    [_, :list | _t] -> "a list with at least two elements where the second element is :list"
    _ -> "something else"
end
# "a list with at least two elements where the second element is :list"

case a do
    [_] -> "a list with a single element"
    [_, _] -> "a list with 2 elements"
    _ -> "something else"
end
# "a list with 2 elements"

case a do
    [_] -> "a list with a single element"
    [:b, :something_else] -> "a list with 2 elements, :b and :something else"
    [:b, :list | tail] -> "a list with 2 elements and a tail"
end
# "a list with 2 elements and a tail"

tail
# []

Since lists are made of cons cells, this matched on the last branch, because although we only had 2 elements in the list it technically is 3, since a proper list always ends in an empty list itself. So by having a list with 2 elements what we actually have is:

(element_1 . points_to_element_2)(element_2 . points_to_end_of_list)() <- this last element is itself an empty list.

If we did this, on the other hand:

case a do
    [_] -> "a list with a single element"
    [:b, :something_else] -> "a list with 2 elements"
    [:b, :list, _last_element] -> "a list with 3 elements"
end

We get instead: ** (CaseClauseError) no case clause matching: [:b, :list]

So although the last element is present, given that it is the list termination (empty list) it doesn't count as an "actual" element, but just as the termination of the list. Notice that instead of separating the last element with | like we did previously, we separated it with a , (comma), effectively indicating our pattern required 3 actual elements, and not 2 elements and a tail. Because the empty list that signals the end of a list is not itself an "element" it didn't match, but when we use | it does match, because the tail might be a cons cell or the end of the list itself - the | separator means the "next pointer" of the list, while , means an actual element.

We also see that because no branch of the case could match, that we got a CaseClauseError, which just means, given the expression passed on to case, I couldn't find any "conforming" branch in those you've defined.

As you see the logic is the same as with the match operator we've seen before, but instead of getting MatchErrors when it doesn't match we get CaseClauseErrors.

What happens when we use case is that the expression in case EXPRESSION do gets matched against each branch of the case statement, so it's translated into something similar to (for illustration purposes):

# when `a` is [:b, :list]
> case a do
>>   [_] = [:b, :list] -> ....
>>   [:b, :something_else] = [:b, :list] -> ...
>>   [:b, :list | tail] = [:b, :list] -> ...
> end

So the first branch [_] (list with a single element) doesn't match, neither does the second branch [:b, :something_else], but the third does, because a is effectively in the form [:b | [:list | []]]. Or [:b, :list | []]

We can see some more examples, now with binary matching.

date = "01-02-2019"
# "01-02-2019"

case date do
    "01-02-" <> year -> "it's year #{year}"
    _ -> "not sure"
end
# "it's year 2019"

case date do
    <<day::binary-size(3), month::binary-size(3), year::binary-size(4)>> -> "it's day #{day} in month #{month} and year #{year}"
    _ -> "don't know"
end
# "it's day 01- in month 02- and year 2019"

case date do
    <<day::binary-size(2), "-", month::binary-size(2), "-", year::binary-size(4)>> -> "it's day #{day} in month #{month} and year #{year}"
    _ -> "don't know"
end
# "it's day 01 in month 02 and year 2019"

date = "01/02/2019"
# "01/02/2019"

case date do
    <<day::binary-size(2), "-", month::binary-size(2), "-", year::binary-size(4)>> -> "it's day #{day} in month #{month} and year #{year}"
    _ -> "don't know"
end
# "don't know"

So this is way more interesting, because now we can start seeing ways to drive the "flow" of our programs by defining the form the data should have.

When we look at functions, all the same concepts apply, but instead of being inside the body of a function, they're used to define what "branch" of the function should be "called" when passed arguments of a certain form.

Let's create a file somewhere. The following examples will all need to be placed inside that module but we'll omit it in the examples, and then the whole module must be copied to your iex shell before running the examples.

# some_file.ex

defmodule PatternMatching do

    def test_1(:a), do: "Function matched on :a"
    def test_1([]), do: "Function matched on empty list"
    def test_1([element]) do
        IO.puts("Function matched on list with a single element: #{inspect element}")
        test_1(element)
    end
    def test_1([head| t]) do
        IO.puts("Function matched on non_empty list, with head: #{inspect head} and tail #{inspect tail}")
        test_1(t)
    end

    def test_1(element), do: "Function with one non-list argument: #{inspect element}"

end

Copy that into the iex shell and you should see something like:

# {:module, PatternMatching,
 <<70, 79, 82, 49, 0, 0, 7, 128, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 189,
   0, 0, 0, 20, 22, 69, 108, 105, 120, 105, 114, 46, 80, 97, 116, 116, 101, 114,
   110, 77, 97, 116, 99, 104, 105, 110, 103, ...>>, {:test_1, 1}}

Now let's try calling some functions:

PatternMatching.test_1(:a)
# "Function matched on :a"

PatternMatching.test_1(:b)
# "Function with one non-list argument: :b"

PatternMatching.test_1([:b])
# Function matched on list with a single element: :b
# "Function with one non-list argument: :b"

PatternMatching.test_1([:a])
# Function matched on list with a single element: :a
# "Function matched on :a"

PatternMatching.test_1([:b, :b])
# Function matched on non_empty list, with head: :b and tail [:b]
# Function matched on list with a single element: :b
# "Function with one non-list argument: :b"

Now if we move def test_1(element), do: "Function with one non-list argument: #{inspect element}" to be the first function defined (and then copy the module again into iex) and run the same function calls we did before:

PatternMatching.test_1(:a)                                                                            
# "Function with one non-list argument: :a"

PatternMatching.test_1([:b, :a])
# "Function with one non-list argument: [:b, :a]"

Now everything matches the first one, because it has no defined pattern it accepts everything so none of the other functions we defined get the chance to be tested and consequently run.

If we add this one function to the end:

def test_1(argument_1, argument_2), do: "Matched with 2 arguments: 1: #{inspect argument_1} #### 2: #{inspect argument_2}"

Then copy the module again to iex and run:

PatternMatching.test_1("arg1", ["arg", "2"])
# "Matched with 2 arguments: 1: \"arg1\" #### 2: [\"arg\", \"2\"]"

Now because we're passing two actual arguments, none of the others will match (since they have arity of 1, meaning the number of arguments they accept is 1) and since only this last one accepts 2 arguments, only this one will match.

If we do:

PatternMatching.test_1("arg1", ["arg", "2"], "arg3")
# ** (UndefinedFunctionError) function PatternMatching.test_1/3 is undefined or private. Did you mean one of:

#      * test_1/1
#      * test_1/2

#    PatternMatching.test_1("arg1", ["arg", "2"], "arg3")

This basically means, "I'm sorry, I couldn't find a function test_1 with arity 3 in the module PatternMatching". It also hints, if it can find suitable hypotheses, other functions that are available in that module. In this case it can see that we have a function named "test_1" with both arity 1 and 2, so perhaps we might have tried to call that but got the arity wrong.

So by now we have seen some use cases for pattern matching and we've learned a bit about them, they will match the first case that has a conforming pattern, be it a branch of a case statement or a function definition.

This means, that the order in which we define the branches or functions has a meaning as well. And also that they will raise specific errors when it can't find an actionable code path. If we define the branch or function as simply a bound (or unbound, _, variable) then it will match everything.

So our patterns must go from the most explicit to the least explicit in order to be unambiguous and actually describe the flow of our program. In case statements, the whole expression is matched as a single element, while in functions there's also arity to take into account.

One last piece of functionality that we can use in pattern matching is guard clauses. These allow us to extend further our pattern matching capabilities. These guard clauses allow the usage of a subset of kernel functions that are "special", in the sense that they are pure functions and guaranteed to be "fast". So for instance we might want to discern if the value is a list or a map, if it's a map we also want to discern between an empty map and a non-empty map, but in the case of it being a list we don't care if it's a list with 0 or more elements, just that it's a list. We could write it as such:

a_list = []

case a_list do
    expression when is_list(expression) -> "it's a list"
    %{} = expression when expression == %{} -> "it's an empty map"
    %{} = expression -> "it's a non-empty map"
end

# "it's a list"


a_list = [:two, :elements]

case a_list do
    expression when is_list(expression) -> "it's a list"
    %{} = expression when expression == %{} -> "it's an empty map"
    %{} = expression -> "it's a non-empty map"
end

# "it's a list"


a_list = %{}
case a_list do
    expression when is_list(expression) -> "it's a list"
    %{} = expression when expression == %{} -> "it's an empty map"
    %{} = expression -> "it's a non-empty map"
end

# "it's an empty map"

a_list = %{some_key: 1}
case a_list do
    expression when is_list(expression) -> "it's a list"
    %{} = expression when expression == %{} -> "it's an empty map"
    %{} = expression -> "it's a non-empty map"
end

# "it's a non-empty map"

If we switch the last two branches and set `a_list` variable to an empty map we get:

a_list = %{}
case a_list do
    expression when is_list(expression) -> "it's a list"
    %{} = expression -> "it's a non-empty map #{inspect expression}"
    %{} = expression when expression == %{} -> "it's an empty map"
end

# "it's a non-empty map %{}"

Although the map is empty, because the 2nd branch matches any map, empty or not, it's that one that gets evaluated when the expression is a map, and the 3rd branch now has no chance to be tested.

So now let's see some real cases where we can use this.

Let's define a module and structure that holds users and the type of users they are along with their age. We'll also define some functions to work with lists of users.

defmodule User do
	defstruct [:name, :age, type: :regular]

    def count_older_than(list, age) do
        count_older_than(list, age, 0)
    end

    def count_older_than([], _, count), do: count

    def count_older_than(
        [%User{age: user_age} = user | t],
        age,
        count
    ) when user_age > age do
        count_older_than(t, age, count + 1)
    end

    def count_older_than([_ | t], age, acc), do: count_older_than(t, age, acc)
end

And now let's create some users and a list of them:

user_1 = %User{name: "John", age: 25}
user_2 = %User{name: "Doris", age: 30, type: :administrator}
user_3 = %User{name: "Jane", age: 28}
user_4 = %User{name: "Joe", age: 60, type: :administrator}
user_5 = %User{name: "Jelly", age: 15}

list_of_users = [user_1, user_2, user_3, user_4, user_5]
# [
#   %User{age: 25, name: "John", type: :regular},
#   %User{age: 30, name: "Doris", type: :administrator},
#   %User{age: 28, name: "Jane", type: :regular},
#   %User{age: 60, name: "Joe", type: :administrator},
#    %User{name: "Jelly", age: 15, type: :regular}
# ]

count = User.count_older_than(list_of_users, 28)
# 2

count
# 2

count = User.count_older_than(list_of_users, 25)
# 3

count = User.count_older_than(list_of_users, 20)
# 4

Now lets add these functions to the previous module and copy it again to iex:

def extract_administrators(list) when is_list(list) do
    extract_administrators(list, [])
end

def extract_administrators([], acc), do: acc

def extract_administrators([%User{type: :administrator} = user | t], acc) do
    extract_administrators(t, [user | acc])
end
def extract_administrators([_| t], acc), do: extract_administrators(t, acc)

And then run

admins = User.extract_administrators(list_of_users)
# [
#   %User{age: 60, name: "Joe", type: :administrator},
#   %User{age: 30, name: "Doris", type: :administrator}
# ]

So you can see that we called first the extract_administrators with only 1 argument, a list, so it matched the first function definition. This function all it did was call the same function, but now with two arguments, the second being an empty list.

This is a fairly regular thing to do, that 2nd argument (in this case) is what is usually called the "accumulator" and it's a simple way of recursively calling functions and "accumulate" the results of each call. In this case it's used to build a new list with all the administrators we find in the original list.

So this call ends up as (given the list we were working with):

extract_administrators([%User{name: "John", age: 25, type: :regular} | t], [])

Given that the first argument is not an empty list, it can't match the second function declaration. Given that the first cell in the first argument list is not a %User{} struct with the type :administrator it can't match the 3rd function, so it matches the 4th function:

def extract_administrators([_| t], acc), do: extract_administrators(t, acc)

In this function, what happens is, we ignore that first value, and we're only interested in the remaining list.

[%User{age: 25, name: "John", type: :regular} | #head
    [%User{age: 30, name: "Doris", type: :administrator} | 
        [%User{age: 28, name: "Jane", type: :regular} |
            [%User{age: 60, name: "Joe", type: :administrator} |
                [%User{name: "Jelly", age: 15, type: :regular} | [] ]
            ]
        ]
    ]
]

So in this case we ignore _ the head, and we call again the function extract_administrators with the remaining list and whatever is in the acc variable, so, which ends up being this call:

extract_administrators(
    [%User{name: "Doris"...} | 
        [%User{name: "Jane"...} |
            [%User{name: "Joe"...} |
                [%User{name: "Jelly"...} | []]
            ]
        ]
    ], [])

Now when this function is called with these new parameters it will actually match the 3rd function clause

def extract_administrators([%User{type: :administrator} = user | t], acc) do
    extract_administrators(t, [user | acc])
end

So here what we do is bind user to the element, and then call the function again with its tail, while also adding the user into the accumulator in the last argument.

You can see that it's a bit of symmetric operation when the head matches our constraints/pattern, we take the head from one list, and we place that as the head of the accumulator, then we pass the tail of the list, from which we took the head from, as the new list to the function.

The first time, acc was empty, [] If we declare [user | acc] , what we're declaring is:

[%User{....} | []]

If we do it again with another user

[%User{} | [%User{} | []]]

And so on.

So now it will be called as:

extract_administrators(
    [%User{name: "Jane"...} |
        [%User{name: "Joe"...} | []]
    ], [%{name: "Doris"...}])

Which again will match only the 4th function clause.

So then it's called as:

extract_administrators(
    [%User{name: "Joe"...} | 
        [%User{name: "Jelly"...} | []
    ]], [%{name: "Doris"...} | []])

Which matches on the 3rd, so it will now call as:

extract_administrators(
    [%User{name: "Jelly"...}] | []],
    [%User{name: "Joe"...} | [%{name: "Doris"...} | []]])

Which again matches only the forth it's called again this time with an empty list as the first argument. And now because the first argument is empty, it matches on the 2nd function clause, where we end the recursion and just return the acc argument, leading to:

[%User{name: "Joe"...}, %{name: "Doris"...}]

These are also composable, for instance you can use:

list_of_users
|> User.extract_administrators()
|> User.count_older_than(28)
# 2

The last part of this write up is about using function clauses with different arities and pattern-matches in anonymous functions. There are a lot of modules in the standard lib that take functions as one of their parameters, specially those that deal with collections, such as the Enum module.

The Enum module has one function called reduce/3 that basically is an abstraction over what we did we these functions. It takes an enumerable, an accumulator and a function, and we can use it to reduce the elements of the enumerable into whatever accumulator we want. We could write the extract_administrators function as:

def extract_administrators(list) when is_list(list) do
    Enum.reduce(list, [], fn
        (%User{type: :administrator} = user, acc) -> [user | acc]
        (_, acc) -> acc
    end)
end

You can see two clauses on the fn declaration. Enum.reduce passes one element at a time from the enumerable provided as the first argument, to the anonymous function, along with the 2nd argument as the accumulator. The first time the anonymous function is called, the accumulator is the original accumulator in the 2nd argument (an empty list), and on the following ones it's whatever the anonymous function returned.

Since we receive each element one by one (outside of their original list) we just pattern match on the element. When all elements from the enumerable have been enumerated it returns whatever is the value of the accumulator.

There are plenty of use cases for recursive traversal of collections, in functional languages that's usually how you work on collections of items. There are also other useful functions in the Enum module. I use reduce and map a lot but there are more.

There's also other tricks you can use, such as using the same name for variables, underneath this makes the pattern only succeed if all instances of the binding resolve to the same value.

So for instance let's say you wanted to take pairs of users out of that list, that shared the same type of user.

def pluck_pairs(list, type) when is_list(list) do
    pluck_pair(list, type, {nil, []})
end

def pluck_pairs([], type, full_acc), do: full_acc

def pluck_pairs([%{type: type} = user | t], type, {nil, acc}) do
    pluck_pairs(t, type, {user, acc})
end

def pluck_pairs([%{type: type} = user | t], type, {previous, acc}) do
    pluck_pairs(t, type, {nil, [{previous, user} | acc]})
end


def pluck_pairs([_ | t], type, acc), do: pluck_pairs(t, type, acc)

Or using the Enum.reduce form

def pluck_pairs(list_of_users, type) do
    Enum.reduce(list_of_users, {nil, []}, fn
        (%User{type: ^type} = user, {nil, acc}) -> {user, acc}
        (%User{type: ^type} = user, {previous_match, acc}) -> {nil, [{previous_match, user} | acc]}
        (_, full_acc) -> full_acc
    end)
end

(we need to use the ^ pin operator to pin down the value of type, otherwise it would be re-bound during the Enum.reduce and match anything)

Another thing to keep in mind is, if the order of the accumulation matters, that after reducing a collection the accumulator will have the elements in the inverse order, so if that's relevant you need to reverse the list, like with Enum.reverse/1 or the erlang :lists.reverse/1 (if it's a lit obviously) function.

And now you could use this inside a case function

case User.pluck_pairs(list_of_users, :regular) do
    {nil, [_|_] = acc} -> 
        IO.puts("No unmatched user and #{length(acc)} matched pairs")
        Notifications.send_pair_emails(acc)
    {%{name: name} = no_pair, [_|_] = acc} ->
        IO.puts("Unmatched user #{name} and #{length(acc)} matched pairs")
        Notifications.send_no_pair_email(user)
        Notifications.send_pair_emails(acc)
    {%{name: name} = user, acc} -> 
        IO.puts("Unmatched user #{name} and no pairs")
        Notifications.send_no_pair_email(user)
        acc
    {nil, acc} ->
        IO.puts("No pairs and no unmatched users")
        acc
end

And so on. Given the natural support for concurrent and parallel processes in Erlang & Elixir, the usage of message passing and so on, pattern matching becomes even more useful, as it's fairly straightforward to describe state-machine'y behaviours using a combination of processes, receive blocks and pattern matching. Of course, most of that is already quite abstracted into higher OTP constructs such as gen_server, gen_statem and other friends. Hope this post helped you understand better pattern matching and illustrated some use cases, although there's plenty more that can't simply be covered in detail here - nonetheless try out and experiment!

]]> https://micaelnussbaumer.com/meditation-without-mambojambo/ Thu, 08 Jun 2017 00:00:00 UTC https://micaelnussbaumer.com/meditation-without-mambojambo/ I also don't like much of the mambojambo that goes around many of the nowadays trends, but I'm an advocate of meditation. I'm going to try to explain why meditation is valuable in a plausible manner, without metaphysical properties.

I don't derive income on the topic, nor anything related - although I don't think there's a problem with selling your knowledge regarding this if you're doing it correctly and helping other people.

Even though I'm going to argue for analytical reasons, you should notice that most traditions aim to realise the "Truth" and answering the most fundamental questions of our existence and reality and in those realms that are highly spiritual we don't have, luckily or unluckily, a scientific language, so terms are used loosely (or everybody has their own definition so globally they become loosely defined, although in each one's definition they're very concise) to describe many things, sometimes not correctly. I'm also not going to argue for its "health" benefits, because sincerely, if you only care about health go to the gym.

Now bear with me, and you might find something worth taking your time thinking about.

How do you come to have the mind you have today?

By "mind" I mean what seats underneath your skin and from which you as a person sprout? Through your words, modes of being, actions, choices, options. This thing that controls these things, is what I'm calling "mind" here.

As your brain developed, even in the womb of your mother, it started being shaped. Experiences, nervous stimuli and others, experienced by your mother, are received as inputs and help structure your neuronal network and nervous system, even before you're out in the world. Which is nothing short than awe-inspiring by itself.

Later, already on the outside, the inputs you keep receiving (even those received in a time when you can't create memories out of them), "print" certain neuronal structures in your brain.

As you grow up, these structures are either re-enforced or not stimulated, and that makes them either grow bigger roots in your mind or wither away, pretty much like a tree in regards to being watered or not. Besides that, new structures get printed, or to say it more poetically a tree of neurons starts growing, with branches, leaves and all sorts of tree like things.

Now lets make a stop to consider what is a thought. So say for instance, you're afraid of dogs because when you were a child, you were bitten by one (I know it's a cliché, but exercise a bit of suspension of disbelief). This traumatic event created a structure in your mind right?

It printed some sensorial information, like a dog shaped thing, maybe the barking sound, maybe smell, maybe many other variables that were involved at the moment of the happening. You didn't specifically record "that" dog, because we humans don't do that, instead you recorded "inputs" that correlated to that experience but, since we humans also have language, you can symbolise these "inputs" and call them "dog".

Any experience creates or organizes neurones in a certain pattern that reflect the experience, in the brain. So when faced with something that fires up one of the neurones (or many) linked to this experience, it creates a certain pattern of reaction in your mind, that trickles down from your mind, to your body, in the form of salivation, an alert, a flight or fight response, a panic attack, a smile, a burst of joy, god knows what. In the case you're scared of dogs, when you see or sense a dog you'll react accordingly.

This is also known as "classical conditioning" and the most famous example is probably the work of Pavlov, where by serving food to his dogs coupled with a non-related input (the sound of a bell, or a metronome) and repeating this action for a few times he got his dogs to salivate just by reproducing the sound, even when no food was being given. Why? Probably because the repetition created a pattern in the dog's brain, where the sound and the idea of food where interconnected so, when the sound was heard, it triggered the same reaction (salivation) as when food was given.

You see, a thought is nothing in itself, it's an arbitrary arrangement of neurones that encapsulate something external to our minds, in a way that we can relate to it and "think" it or "experience" it. It can also trigger other patterns and groups of neurones and make them fire.

Usually in our lives these "thoughts", or patterns of neurones, are fired in specific situations. You can become "aware" of them, say for instance the dog situation but, in many others that aren't as clear as that, you might not.

In the case of the dog some people will say that, to overcome this irrational fear, you should try to spend some time with dogs that aren't aggressive. By doing this you are probably simply replacing by repetition a certain structure of neurones by another one. If you do it several times, when you face a dog instead of the experience activating the old traumatic events, it will activate these new made structures, that you created by spending time with them. So the "input" of an "object" that has the characteristics of a dog in your mind no longer fires the neurones, or the pattern of neurones, that would trickle a fight or flight response previously.

So this is cool right? It kinda means that you have the power to shape part of your reactions - you could even call it, the power to hack your own mind.

Now, like I said, most of our thoughts and reactions occur in specific situations but we aren't always aware of what actually triggers them. Some times because we are ourselves experiencing the situation, others because it's not always obvious to us, and yet others because we simply don't have the time to go through the sequence of experience in a detached manner (or we may not even be able to).

This is where meditation fits in.

When you start meditating, what gets fired in your head is at the "surface" of your conscious mind (you see, how can we talk scientifically about this shit really? surface of the mind? is that even a thing…).

Because you haven't trained your mind yet to have an anchor, all the electrical impulses firing through your beautiful scalp trigger all sorts of day-to-day thoughts that are in your mind and, if you're anything like other people, you can probably see that the thoughts towards which you're drifting are related to recent events or forth-coming events.

Once you begin to get a hold on how to re-focus your mind from this so said "superficial" thoughts, you begin to go a bit deeper in the machinations of your mind. But to be able to re-focus and sustain focus outside of the thought web you'll first need to practice. Sometimes it will seem pointless but I can assure you, as an over-thinker, that you'll eventually manage it, as long as you keep sitting in silence regularly.

At this level, it's a bit more subtle. Thoughts arise from random impulses in your mind. Where do thoughts come from when you have no external stimuli for them?

I would say that since your mind is always operating, what starts happening is that what gets fired inside your brain - I mean the patterns of neurones that get fired - are the patterns that are more "vividly" printed in your brain. (and they can be vivid both by intensity and intensity together with recent timing, the last usually fading away as time moves forward, since they stop being recent - like if yesterday you had something important on your mind it might pop up but after a week, or even next day if it's no longer relevant, it probably won't pop up again)

It makes sense no?

It's like you have a room full of air and an inflammable gas floating in it as well, in a relevant quantity. If you light a lighter in there - BAM! - it ignites some particles near the lighter and they follow the thread of the gas igniting it as it goes right?

If you don't have any inflammable gas, or not in a relevant quantity, you can light the lighter as many times as you want, that still, nothing will happen.

This is similar to what happens with thoughts in "our mind". If you didn't have patterns of neurones more predisposed to be fired, when you sit still and after you master focusing on an "anchor" (breathing, mantra, or others), there should not be any thought arising right? But they do. And the most funny thing is they do arise besides not having any "situation" firing them. Right? There's no external lighter. So what is making them fire?

Think about this, when you're meditating, and you have gathered a certain amount of experience in doing it, you're able to see thoughts for what they are. Since you have practiced for a bit this you're, actually, able to look at them in very interesting ways. Since your mind is now used to the silence, you can perceive things such as the emotional charge thoughts have, you can see them "rise" from the unconscious to the "surface" of your mind. You can also see what they "fire" as they rise.

We use trauma mostly to talk about something that is "bad". But a trauma is something that leaves a mark. If you were nurtured in good measure, that's a trauma too, it just has good repercussions. The thing is, everything is up until a certain age traumatic - in this sense that it leaves a mark.

If you then add the fact that many of the behaviours and thought patterns we have that make up our own "minds" are made when we don't have any tools to realise we are shaping our minds, or in other words, programming them, we could come to the conclusion that what you think is you is not actually you, or what you think are your original thoughts aren't actually your own thoughts, but the result of repeated inputs. If you then add to it the continuous flux of information to which we subject our minds in our current contemporary societies - even from a very young age, that incessantly "print" inside our minds these structures of thought - the more meditating becomes an important practice.

Have you ever thought why Advertising in prime time was (and still is) so expensive? And why advertisements pass repeated times at the same time frames? Because repetition will print in your brain interconnected sequences of neurones, like a product with a kind of emotion or feeling. Does that ad have a semi-clad girl? Or a happy family? Or the hint of adventure? Well these things are powerful triggers and, when repeated several times, they will print directly in your mind certain structures in certain patterns.

You might find yourself salivating unconsciously for a product that's totally unrelated to the emotion you're experiencing, just like Pavlov's dogs salivated for the sound of a bell (or a metronome).

By meditating you're actually giving yourself a tool to go through this. Literally, instead of having only external "lighters" to trigger your thought patterns you're giving yourself a "lighter" that will trigger whatever are the most "rooted" neurone meshes in your brain (you might think of it in terms of path of least resistance - since all neurones are fired by electrical impulses the ones that have a bigger hold in your brain or more ramifications will undoubtedly fire more easily).

A tool that also allows you to distance yourself from the "thought" objects that were "printed" in your brain. Because they're just "thought" objects.

Your mind can be seen as a store warehouse for "thought-objects" that replicate certain aspects of the external reality, apprehended through your life through your sense organs. This doesn't mean reality doesn't exist, it means that what you have inside your mind is not reality, although in many situations it will be as close to and useful almost as if it were literally bits of reality.

Say for instance you have a child, or something that you really love, when you think of it it probably triggers emotional responses and emotional patterns right? It's not that the object itself carries these emotions or patterns, it's because in your mind the object triggers these other things so, for you, this object is emotionally charged and both are the same. And you can confirm this. Take the object of your love to somebody who is totally unrelated to you and it and just watch them have the blandest of reactions, right? If the object carried these emotions, for instance your baby, then whenever another person entered in contact with him, he would experience the same feelings as you. But that doesn't quite happen right? If your baby starts crying you'll probably be worried and afflicted, but other people in the room will just go - wtf, just shut up that annoying brat.

Once you understand this - and the empiric experience of this is much stronger than the theoretical understanding of it - you're in a way a bit freer to choose things in life, I guess, and to understand yourself.

By exposing these thoughts and the subtle ramifications of the patterns that inhibit or trigger them you get more comfortable with them and, by repeating this process of sitting still and letting your mind unfold, you start to be able to, not control them but, be less controlled by them.

Because even though usually it's useful that the thought objects mimic reality to the point they create a reaction in you some times this may not be as good.

It's good that you're able to discern from an immediate threat, or pick up signals in a conversation that tell you something, it's good that a person you love triggers all the loveable patterns of emotion you have - otherwise you would be a robot - but at the same time, it's not good that some things trigger responses that you wish you wouldn't have - because this too is similar to a robot like experience.

In this sense by practicing you go through several stages, every time you re-focus your mind you're doing a mental pushup.

Some people will also be confronted with some thought structures and patterns they weren't aware of and that may increase some seemingly negative responses. Understanding and acceptance aren't always painless, but not facing that which brings you pain or sorrow will not help you.

Some times it's abstract, some times these patterns of thought are so deeply rooted that they can't be symbolised by language or idea, they just are.

You take a step towards freedom every time you overcome and understand these responses. And then you keep on repeating. Once you start clearing this, you start to settle more often in a specific electrifying state of restful awareness, where less and less you fire these patterns and you're just experiencing the "lighter" going on, maybe what some people call the divine spark? (just kidding…).

The things is, you should try it for yourself and you should stick with it. If you see a guy all pumped up from gym telling you you can do the same, obviously you're not expecting you'll achieve the same results by going to the gym for a month. You need to keep doing it right? I mean, you understand that right?

And so is the mind. It's repetition, either of inputs or behaviours, that shape what we are (along of course with nature), and by giving yourself time for them to unfold you're equipping yourself with a better knowledge of "who" you are. It also touches several aspects of creativity. Creativity is sort of the ability of coming up with new ways of organising or using things. If your mind is held in the sway of certain patterns of thinking it becomes very difficult for it to come up with new ways of organising information, whatever this information is, sound, painting materials, programming languages, tools, whatever.

]]>