Tuo

Midlife Crisis as Programmer and AI

2023-12-23T11:55:32+00:00

Just like an ordinal programmer, I studied Software Engierneering as my major in university. After graduating in 2010, I worked for almost two years at Thoughtworks as an agile consultant, primarily focusing on Ruby on Rails web development and scrum practices. However, I found mobile developement hot and intriguing and decided to give it a shot. I joined a small mobile development team at ReignDesign, specializing in iOS develpment, with some Android as well. Five years later, when the company relocated to Chile, I transitioned to my current role as teach leader in a new company as my ex-colleuge invited.This new role involves hanlding the entire techstack(web/app/backend/infra). What particularly interests me is the challenge of expanding my coding skills while also managing a team.

Each transition has been smooth for me as I’m able to learn quickly and adapt. I still have a passion for technology and coding. While I may not be as eager as someone attempting “Seven Languages in Seven Weeks”, learinng new skills is a routine part of my daily coding, and I thoroughly enjoy experimenting and playing with them.

In 2021, during the period when the FED was printing money like helicopter and every tech company seemed to be expanding and hiring, I was approached by some companies. The first one was Facebook London, and I performed poorly in the coding part of the interview. Well, I hadn’t taken any serious inteviews for a long time and barely knew what the current tech interview looked like. That failing was a genuine shock. I took a step back, quickly gathered my thoughts, condcuted some research, and thoroughly reviewed the interview process. Then I worked on the basics for LeetCode and system design. The Behavioral part was never a problem for me, as I always spoke truthfully about myself :) In about four weeks, I practiced diligently on algorithms and designs in my spare time. Surprisingly, it refreshed some knowledge I didn’t know before and learned some new things.

The second time I tried with Booking.com, and it didn’t go well either and failed at the system design part. However, I became more familiar with the process and gained experience. Later, I succesfully secured two offers from CloudKitchen and Flexport. CloudKitchen was pretty cool for its homework assignment of coding part, but it just looked too mysterious and the vesting plan was not friendly. Flexport wasn’t as appealing to me. I guess tech company need just spend those money out to get more money and their pay was good at that time. Ultimately, considering my familiy reasons and the fact that I wasn’t actively seeking new opportunities, I decided to turn down those offers. When Facebook London approached me for the second time, I replied to them that I wasn’t interested. So, I chose to stay with current company.

What goes up must come down. Towards the end of 2022 and the beginning of 2023, the global economy is slowing down. The big four tech companies are undergoing massive layoffs. It is not just the tech industry; other sectors have also begun to cut their spending. This impact is being felt in China as well. My once-filled LinkedIn message inbox is now almost empty. What’s even worse is that the clients of my current company have either lowered their budgets significantly or completely remove them.

I’m 35, about to turn 36. The reason 35 is a sensitive age in China is that if you want to work for the government and get a stable job, you need take civil service examination, which has an age limit of 35, even if not explictly stated. Therefore, that door is now closed for me.

In a developing country, society is changing rapidly. There have been plenty of young people graduating from college every year since the early 2000s. The unemployment rate for young individuals is high. In fact, in most big tech companies, the intensity of work is extremely high, with a 996 work culture here employees often work from 9 am to 9 pm, six days a week, resembling Japan’s work culture in the 80s.

There is a widespread consensus about “35 crisis” for programmers. You can find numerous posts on this topic discussing anxiety and confusions on the popular Chinese programmer forum: v2ex.com/t 35岁. Some can’t help but asking, “Is food delivery and ride-sharing the final job for elderly programmers?” source:大龄程序员的出路难道是开滴滴送外卖吗 - V2EX. Even though the IT industry has a relatively short history, spanning around 30 years, and dicussions on simliar topics used to popup every once in a while, in past two years, one can sense a certain coldness and chill in the industry.

With the emergence of AI technologies like ChatGPT, even some coding tasks can now be accomplished by AI. The concerns that arises is, are programmers going to be replaced by AI?

Yep, that’s what I’m facing. I might lose my current job soon, and it doesn’t seem like I’ll find another one anytime soon. All those years built on a somewhat great feeling of “I’m a good programmer, and everything is under control” now resemble a ballooning situation that suddenly gets pierced, and the sound is so loud it leaves your head buzzing. It is like a curveball thrown by life, and those feelings of comfort, saftety, and secruity dissoves and vanishes in the blink of an eye.

Damn, that really hurts.

Observations from Kid Playing

We think we understand the rules when we become adults but what we really experience is a narrowing of the imagination. (David Lynch)

There is a ruler from society that measures our outward material success. These metrics are visible, and we’re accustomed to measuring ourselves by them since we grow up. We aspire to climb higher in that ranking by that ruler, believing it will make us feel safer and better. Subconsciously, it becames a part of our ego, and even we identify ourselves with it.

Now, by this ruler, I’m sliding to the bottom part, fearing being eliminated by KPIs. That ego and identity starts working against me, triggering a chain of negativity, connected with some of the worst memories and emotions from the past. This leads to a serious self-doubt. I can feel the urge to desperately try to get out this bottom as soon as possible.

That self-doubt has prompted me to review my whole life.

I start to ask “Who am I? Did I really love what I do ?” What if that ruler is gone, “who am I really?” To what I should measure myself against?

In recents years, some of my best family relatives who see me grow up passed away abruptly. They passed away in a way I never thought or expected. I ‘m so used to thhat they are gonna be there forever, and now suddenly gone. Life is uncertain; It leaves you pondering on how fragile life is and how small humans are. Hence, I want to see through that external ruler, e.g. material possessions, social norms, and strip away those superficial layers, and put aside labels like “practicality and maturity”, just to get a glimpse of that in its raw form. One thing clear to me is that is not about meeting other people’s expectations, not for vanity, not for fame, not for money, not for comparison. None of that. I have an intuition, a hunch that the answer lies not in the outward, but in the question itself - inward - deep inside me.

I’m bothered. Life goes on. I have an almost three-year-old kid. My wife and I live away from our hometown, renting a house here. We play the roles of “teammates/comrades”, as we call it in China here, with the same goal of taking care of our child. My kids tends to stick to his mom all the time. When my teammate needs her own space, like taking a shower or going out shopping, I have to ensure I can took after the kid without leaving him crying for mom. It’s a challenging task; I need to distract him. Well watching his favorite shows like Pepper Pig, Bamboo Club, Pleasant Goat and Big Big Wolf or dinosaurs won’t last long, so I have to actively engate with him via soem activities. Sometimes, I take him to children’s playground indoor/outdoor, some park, and sometimes we just play with toys at home.

It was one of those ordinary moments; my son sat there playing with his building blocks, and I lay beside him, just watching him play. I could see he was fully engrossed in the game, his genius and dedicated expression on his face, as if time had frozen, and the world around him had blurred out. I felt something drawing me in, resonating with something once familiar and deep - the serenity and joy. It just healed, and in that space and time, I was free from all kinds of anxieties, fears, and desires. I don’t want to sound corny, for example, religious revelations - like Martin Luther’s caught in a thunderstorm and Blaise Pascal’s two accidents of riding in a carriage - I did feel at that time I saw some light glowing from his background. That Aha moment awakes me in a way of non-logic/non-rational even inexpressable, and my initution was right.

It reminds me of those similar sparks I had during playing basketball and fishing. In some intense situations, I found that if I was able to drop out my pre-plans of my “best” moves to attack my opponents, suddenly my mind looks like it hasn’t got a chance to spin up and plot some plans. It is a natural and instinctive move just coming out of me. That move is totaly unexpected yet smooth. The oppnonet is like a dance companion. We dont’ figiht against each other; instead we dance togother. There is not a “me” in this picutre; the hoop, the oponent, and myself are just harmoniously blended as one. I’m so concentrated, like my kid is playing; somehow, the line that divides and differentiates people and objects seems to be gone. I remember I keep recalling that unfathomable experience all the time.

I also notice that when my son is playing, he is free from the fear of being judged by what the final result should be, the rules instilled with how it should be played, not to meet somenoes’ expxectation or approval, above the duality of good/bad and past/future. That ‘s something that we can all relate to, that’s who we were in childhood. Kids know how to play, they’re born with the gift to just play, to explore, to discover.

Maybe we think they haven’t develped a high-level consciousness , hence only with subconscious and aninmal spritit, but sometimes people’s consciousness have gone too far and burden me so much as I grows; it just overwhlems other aspects. I forgot that play is just simply for sake of play.

Here are three takeways from my observation:

Blank paper or empty cup mindset: To play is to leave behind the impressiosions or preconceived ideologies, to engage in play to disover and explore.
Switch play smoothly: When the play doesn’t seems fun to him, he git-stashes it and switches to a new play naturally. No baggage or burden of the previous play is carried to next one. Itis about single-tasking, not multi-tasking.
Playing is a nature state of being: Not to please or seek benefits, nor to escape punishments. It’s a pure and unfiltered state of being.

My Experiement

In the beginning, I really wanted to get rid of that anxiety and self-dbouting, but soon I found myself in a position of choincelessness. I couldn’t do anytihng about it. That has made me to slow down involuntarily. Anyone could feel that slow motion of time when invovled in negative emotions. With the playing experience from my son mentioned above, I started to play with awareness - trying to apply that in different aspects of my life through experiments.

Quite the oppose of the scentific side, I checked my criticism, logical analysis, and discursive thinking at the door,attempting to connect again with he once-swept-under-the-carpet child innocence. To have a playful mindset, to release the inner child. I’m not meant to be an anti-intellectual. When driving and wating at the crossroad for green light, you can’t say, “ Hey , red is not red, green is not green so let’s go”, as it would not only hurt you but also hurt others. However it would be perfectly suitable for me to walk in the park, look at the green leaves and red flowers, and conduct an experiement of dropping that “red is red, green is green” judgetment or concept or categorization. Just see the things behind its names and labels, feel their essences and reality in a direct way. It is at my disposal, and anytime I could stop and revert.

One of those experiments is about trying to distract my sons’ attenion from his mom. I saw a couple of times when he had a pen, and he just freely drew by himself. I thought, why not do some basic idiot-proof hand pencil drawing together? Well quickly I could hear a voice inside my head, whispering, “Screw you, just recall, throughout your life, you have no aesthetic cells at all, no taste. Come on,you’re a programmer, not a designer; you are just good at playing 0011s. You dont’ that have crazy hairstyle or unique looks. No artistic temperament. Don’t be childish”. Yeah, but this time I thought “ Whatsoever, I’ll give a shot”. We started drawing first with his favorite animals. Sometimes, he would collect different leaves from trees to show to his mom, adn later, we would sit together and just oberve the texture, lines and sahpes of those leaves, and draw them on the paper.

It is a clicking experience - two human beings are playing, sharing and connecting. There are some vibrations.

Those drawings, by any academic standards or social standards, wouldn’t be considered art. But I feel it is the art, as for me , art is about epxressing oneself honstly and being authentic, showing what one sees and feels in life with their heart. Art is not fixed or limted to some labels, titles or jobs; it is limitless and is everywhere in everybody.

Creativity

As I build up those expeirements, one thing starts to emerge and keeps hitting on me. That’s when I do one activity, for example, Active A, and I bump into a wall, something like writer’s block, where you feel you have nothing to say about and no clue how to move on. You’re kinda in a mess and have no inspirations, like a stream that gets drained.

Instead of forcing it with my will the hard way, to keep banging at the wall, I notice life leads me to next thing with a smooth transition. It is when you got stuck; maybe the sun is shining outside, so you go out for a walk; maybe it is raining, oddly enough, and I want to feel how subtly the raindrops fall on my face, like what every child would do for the first time; maybe it is lunchtime, and I go out to grab some food; maybe my kid invites me to play, and I play; maybe friend asks me out; maybe I’ve been sitting too long, and my body reminds me to get some exercise, like fishing or playing basketball or jogging or streching.

I know it sounds conry, cliche, and very subjective, like a faith healer from Megachurches. I do feel It always offers something or activities from different parts of life to move on to next.

I stash those things about Activity A, and go with fully empty mindset like a blankslate into what comes next. Later when I came back to Activity A, to my surprise, most of the time, I just have some different views spontaneously. It wont’ directly solve my problem at hand or guarantee to bring inspirations, but I found it interesting to play with those views, and usally, it gives new dimensions that I never got into before just free to explore.

It seems nature is sorting through those raw, unfiltered gathered pieces of information in the background without our knowing. It merged the new with our existings knowledge like an alchemist, connecting different dots and aspects of life, much like a child’s drawing and exploring, in a mind-blowing form. Then, all I need to do is just: “Trust the Process” .

But sometimes I think: “How you do know it? How sure you are? What’s your proof” ?

Chuang Tzu who is a classic and famous Taoism philosopher, dating back to 530BC in China, put it very nicely:

Those who rely upon the arc, the line, compasses and the square to make correct forms injure the natural construction of things. Those who use cords to bind and to glue to piece together interfere with the natural character of things… There is an ultimate reality of things. Things in their ultimate reality are curved without the help of arcs, straight without lines, round without compasses, and rectangular without right angles… In this manner all things create themselves from their own innermost reflections and none can tell how they came to do so (Ch. 8).

I don’t know, it is just my gut feeling. It goes beyond conviction, faith, belief. I’m giving it all and just aligning and flowing with it, with no expectations or attachments. It is immanent by ones’ nature.

I will do my part by trying to keep my window open, setting up my antenna high on the roof, just to pick up any signals that nature is sending to me. At the same time, I collect unfiltered informations from expericences in my life in fullest potential and “send” it back. Nature is doing its part by sorting all the information I just recieved into the right places. I dont’ borther about my originality, nor try to preconcevie it or to control it by any efforts and means, as any form of interference would do nothing but destroy it. I just do my part, and the brain and nature takes care of the marinating process, thereby taking care of me. Instead of control, I surrender. It is like a tacit agreement. It is like showerthoughts, you couldn’t figure out how it just pops up in your head, but it just happens.

The abliity to connect different parts, things seemingly unrelated on the surface, is the nature that has greatest and most primordial source of creativity.

I could get in touch with it through the lens of expericence of the diversity. Those gems are scattered in the most mundane of our daily life.

Many artistists claimed to have had experiences using drugs as a source of inspriation and creativity, even some the biggest names. Well, here’s what I think: when you’re in a state of craving: “it feels good, I need more; I have a tight schedule, deadlines is looming, I need get it done i before xxx, so just give it more; I need sold 2 millions copys; oh I need it to get into top 100 of billboard; oh this one is good, I need make next one more impressive”, trying to force your will upon it literally closes all windows and pushed that connection away. Some take shortcuts, especially in easily drugs-accessible environments, desperately using drugs to throw themself in a delusion, hoping to get what they fantasize about. Eventually, the shortcut takes its toll. Not only does it lead to the loss of sobriety , but it also damages their body and brain in inreversible ways. Most importantly, it breaks the tacit agreements. They might have lost the connection through other ordinaly life experiences. The threshold gets lifted up and not easy to fall back to normal. The relap is bound to happen.

Creativity’s emergence is not intentional but natural and spontangeous. Sometimes, thinking is the biggest mistake one can make, one has to feel and touch with ones’ raw sense.

The inner boundary and inner compass are something we, as human beings, are born with by nature. However, we gradually lose it during process of the growing up. It bring peace, tranquility and freedom. The boundary is set up voluntarily and spontaneously, not baed on one’s desires or will, nor dictated by societal rules. I’m not suggesting exploring the flaws of laws, like what Wall Streets and Crypto world (except Bitcoin) often do, while the laws and the moral code from outside may become stale, rigid and outdated, but one’s inner code and law is always fluid and up-to-date.

What connects with people is you connecting yourself

I need to clarify a little bit here: the things I referred to above, such as the essential truth, fundamental principles, objective realtit, natural, universe, or the alchemist -whatever you name or label - all points to something beyond.

The thing behind is something couldn’t be put into words. It might be interpreted as an intuition of “Oneness”, “Void”, “Emptyness”, “Unity” or “Substance”. But those would limit it, as if it is only related to religion. It is beyond human language yet not isolated from our daily life. No complicted jargons or fancy spritual words need to be put here. Anyone or anything that looks up to the sky right above them could get it. Therefore, the word “universe” is more common and relatable for everyone, regardless of their background, ethitiy, color or nationlitiy, and will be used in following part of this article as reference to that inexpressible, unfathomable, incomprehensible, infinitely elusive thing, not by its literal meaning in science.

There is an ineresting story from Chuang-tzu, which can be found in his work : <A Happy Excursion> .

而独不闻之寥寥乎？山林之畏隹，大木百围之窍穴，似鼻、似口、似耳、似枅、似圈、似臼，似窪者，似污者；激者、謞者、叱者、吸者、叫者、譹者、宎者、咬者；前者唱于，而随者唱喁；泠风则小和，飘风则大和；厉风济则众窍为虚；

It literally says: There are thousands of thousands of holes in the mountains with different shapes. Some look like a nose, some look like a mouth, some look like an ear, with different internal structures. When the wind is blowing on the earth, either small or strong, those holes generate different sounds, each with a unique pitch and tone. In whole, it is as if nature is conducting a symphony or orchestra. How harmonious it is!

Each hole has a different internal structure—some may look messy, and some may looks perfect. However, when each hole is self-asserting, each contributes its own unique tone to that grand melody. Despite their differences and diversity, each has its root in the shared harmony of universe. These individuals are somehow connected at their deepest level. When each is expressing themselfves, quite counterintuitively, they don’t have a specfic aim for that harmony, or it doesn’t exist in their heads; it is never part of their initial drives—it is all about themseves.

Suprisingly, when one expresses their seeing and feeling about the universe through his unfilterd lens in their own way, the universe seems to better connect to others. It awakens the true authentic side of someone, inviting them to make their own sounds—no pretentiousness, no imitation. Hence connecting as a whole and feeling the flow of harmony.

Don’t seek to copy what someone does; dont seek them—seek what they’re seeking, what they are after. Don’t concentrate on the finger; instead, focus on what it is pointing to—the moon, the universe, the harmony—the one possessing the greatest source of creativy, beauty and compassion.

Midlife Crisis

The midlife life crisis, if I don’t identify myself with it, If I just leave the ego part aside, could be an interesting thing to look at. The overwall evironment is not friendly in tech market; at the same time, I need provide bread on the table, and my energy and body condition are decling—those are the realities with no denial. To flip the coin, there are some implications：What if career is not about some fixed path? Waht if finanace is not about linear progression? What if body conditionling is decling but becoming more efficient?

Life awalys providers you something new. And those are great times to challenge some long-held ideas, assumptions, or impressions. To peel off the ego, artificallity, and prentence, to engate in a direct, face-to-face conversation with the naked self. Then it is just a play in a different form.

Anyone who tries and pratices basic punches would understand that, instead of just focusing on using fist or hand alone, it is actaully a flow and sequence involving pretty much the whole body. The power starts with legs from the groud, then the hips and core, back and shoulders, and finally to your arms and fists. Different msucles engage and do their own work; they know when to be flexible and when to be focused. As a result, they provide maximum power with efficient body mechanics. The body seems to know how to corrdinate and express itself in its nature and spontaneous way as one whole, indispensab, and indivisible. This is the same for shooting a basketball. The majority of power to throw the ball comes not mainly from the arms and hands, but rather from legs ,hip and core.

The world is your oyster.

In sports, I started to do experiment, committing myself to let my mind recess and letting the body reveal itself, to be the protagonist. Trusting it and letting it work in its own way. With more awareness and pratice, I found that I’m able to get into that flow/zone state and have that lazer focus more often.

That’s a hell of a lot of fun in this process.

Well, don’t get me wrong. I’m not saying we shouldn’t do planning, but that planning itself requires a prerequisite to living fully in the present moment. Therefore, not being attached to the outcome of the planning, not identifying oneself with it. Anytime, anywhere, we are playing, experiencing, flowing, and being. At any moment, we just know that we are self-sufficient by ourselves.

Just think about it: the midlife crisis is actually not that big crisis comparing to following one. In this digital era, the Brave New World, where tranditional values were wrecked, there is a huge void and emptness at the heart. Some products, developed with sophisticated algorithrim and designed with attractive intefaces UI/UX, are built to solely exploit the dopmaine of theirs users and get them hooked. Those are the most intellectual people with a very deep understanding on the science of psychology, the dopamine system, and how human brains work. These products keep someone busy with instant gratification so that they don’t have time to think about that void and lonelness. Another would be to create an image of ideal self on the internet and get other people admiring or envying, all shiny, bling-bling, facny life. It makes them feel good, and it seems to be a sign indicating someone has found their success.

In a world where flooded with so many stimulis and easy-pleasures-gainings products(source: Thanks to the Internet, I have probably seen more naked ladies than all of my ancestors combined. : Showerthoughts (reddit.com)), to find ones’ real vocation, one can’t rely on one’s will and seek some magic silver bullet from the outside. Because that outside relief is like spring, the more you press it, the higher it would bounce back, hence hurting more. We need connect to something bigger than us—the universe. We’re universe inside a human, not a human inside universe. From the attainment of that state of being and high-functioning, universe would support us by all the powers and potentialities inherent in its diversity and vitality, hence one’s strength would go far beyond what ones’s will and ego-form self could offer. That force and empowerment could help one gain clarity through the foggy smoky cloud haunting us, and find themselves a serenity, peace and clam state. That source of strength starts from the center of one’s being.

The nosier it is, the more obvious the quietude would be. Those strong negative feelings and emotions from the so-called “crisis”,maybe—I feel—is the desperate awakening call to some part of my rigid/fixed way of living. In that moment, I have never felt so aliving before, as I realized I had been dismissing and rejecting it.

Not too bad, fun to play.

AI

In the beginning of 2023, along with ChatGPT, there are LLM based code interpretors such as Microsoft’s Coilpot. I use both of these extensively in my daily coding. They help eliminate tedious and boring trivials coding tasks, almost remove the need for searches on stackoverflow or google, and significantly reduce the cognitive workload.

Tools like fine.dev and sweep.dev even enable product managers to make some junior developer-level changes through a chat interface. Those tools read the codebase, plan the changes, write github issues, and submit pull requests. For instance, if a product manager needs to change some text or colors - those small changes, they can directly tell the AI and generate code changes and even auto trigger the deployment process on CI system without involving a developer. Although it is still somewhat limited and can only handle simple and basic codings. And the demo might look fancy and impressive but not perform well with real-world projects, it certainly showcases some promising potential.

There are concerns that if AI can do some junior developer coding, it may eventually progress and evolve to senior developer coding, posing a potential crisis for programmers who could be replaced.

This brings to mind the Luddite Movement in early 19th-century England, where textile workers sabotage the emgering machinery. Those workers belived that the machines would take away their jobs and perceived them as threats to their livelihoods. There was a strong fear that the introduction of this technologies would lead to the displacment of traditional skilled workers, declining wages, and worse working conditions.

As industrial revolutions advance and pipelines are introducted, the workers has turned into “screws” in the machinery of the product line. In Charlie Chaplin’s 1936 film “Modern Times”, the portray of of small individuals confronting gaint machines shows the repetitive and dehumanizing nature of work. Humans are percieved as nothing more than tools and instruments for the end goal, rather than being valued as human beings.

Man is a slave to the machinery, much as it was a slave to the God before the Humanism Movement of the 15th century.

Before the Humanism Movement, the prevailing belief was in the fallen and sinful nature of human beings from Middle Ages. Remeber what St Augustine saids – three vices of lust, there is a disease called curiosity. Humans were expected to always remember to be humble by their sins(pride etc) and consicious of their miserable and despicable existence, watched over by a patriarchal God. They deserved it and should live in fear of the Law, at the mercy of God’s grace.

“From the same motive,men proceed to investigate the workings of nature, which is beyond our ken—things which it does no good to know and which men only want to know for the sake of knowing.” (Saint Augustine - Confessions)

However, people could be gawking at freaks in a sideshow, meanwhile, they could make investigations in nature/universe and comes up with philosophy like those great classic greek figures did. Pico Mirandola, in his “The Oration on the Dignity of Man”, boldly and loudly declares, that human beings have freedom, are marviersous and splendid creatures in their own right. Therefore, as human beings, as individuals, they should focus on their value and free will.

“We have made you a creature neither of heaven nor of earth, neither mortal nor immortal, in order that you may, as the free and proud shaper of your own being, fashion yourself in the form you may prefer. It will be in your power to descend to the lower, brutish forms of life; you will be able, through your own decision, to rise again to the superior orders whose life is divine” (Mirandola, 8)

How strikingly similar it is. Human beings are above machinery, above tools, and, of course, above AI.

The fundamental aspect is this: as a human being with free will, how can we explore and discover something unqiue about ourselves? Something with emotional depth, authenticity, genuine creativity, and originality by working his own way, instead of trapped in self-made rat race.

As a developer drawing from my 13 years of experiences, by observing both myself and other devlopers from the trenches, I’ve noticed that some developer(include me) can’t avoid falling into the trap of identifing too much with a specific language/techstack. To learn something new, even totally contrary to what someone knew, it would be a process like someone can’t fully listen when somebody speaks, because their mind is cluttered with inner chatter and jugements. It is like a cup full of water, you can’t fill in more water. Similary, the mind has become jammed and is not able to be flexiable and fluid to in taking in new stuff, slowingly becomes dead water, resulting in stagnation and fixation on technical approaches.

A programming language is worth learning if it is different enough from all you already know to challenge your mind, exposing you to new abstractions. That reminds me of Closure programming language when learning Java back in early days.

To a man with a hammer, everything looks like a nail.

Coding is not just about being alone with computers; it overlaps with one’s other parts of life. The code developer write definitely need get a good understanding of machine and its fundamentals - how operating system(CPUs), kernel, system calls works - and make best out of it by using algorithm, tool or design, while also the logic should be simple and straightforward for people to understand. For example, common pattern would be to sacrifice space for time, introduce redundance for speed. When to take compromise or not. And the ability with experience to identify fundamental feature and a non fundamental one, to put designer’s shoes, to think in a bigger scope, actually saves a lots time and unnecessary complexity. One can’t preplan too much at the beginning, at the same thing, one can’t afford to do no plan/design at all with his experience and logical analysis.

Counterintuitively, writing the code to implement is somewhat the easy task. The hard task is understanding what to do and in what way to do it.

It’s also about balance, more than just logic—it’s an art. To be agile is to embrace changes, cultiviate a hunch knowing when to rely on veteran experiences and when to think outside the box. One common scenario is that when we write code, to some point, we got stuck.Much like creative progress, I found one could try following tips: First, pull yourself away from that little screen to explain to colleagues or someone else. This forces you to step out of whatever context you’re in, think from a high level, and start with the basics. Second, go out, take a walk, have a cup of tea or coffee, do some stretching, etc., and do it wholeheartedly, without the worries or tanglings of the previous problem.

It’s possible that someone could gain different perspectives when returning to the desk. Many times, when I try to articulate my problem and what’s my stragtegy, suddenly I understand why. Give a try to trust that the universe and life could shed some light somehow. So, when you feel stuck, try pausing and stepping out of that space to try something different.

Programming is art in certain way.

I believe that programming and life is intricately intertwined. No one could just keep consuming, and leave no time for digestion; No one could just keep multitasking, and leave no time for singetasking. In order to get most out of our reasoning, logic analysis and decision-making, one seemingly unconventional approach is to suspend those desires and thoughts temporarily at times. I mean, to create a gap, a space for it to let the analytic mind rest and let raw ability of connection to reflect itself. It is in that space, for instance, times of doing nothing, where by social standards our behavior it might look childish, non-productive, weird and meaningless, the mind won’t be kept bombarded, and brain has time to cleanup the working memory, and universe has a chance for its magic to happen, therefore keep a high-level of efficiency and quality of our congnitive function and be a state of inviting the peace and creativity.

With that said, If someone have one’s mindset prepared and got some content someone want to express, the cutting-edge technology like AI can open up numerous possibilities. Prompt engineering allows tweaking input for detailed and sepcific request, treating it as if assigned a particular role to get the optimal results. Asisstants finne-tune and transform into your personal agents, while tools like AutoGen and LangChain extends LLM with external components, making creation of LLM-driven applications more conveniently.

(ps: Every time, I try to write my first shitty drafts on paper, so that I can revisit and make adjustments. When I type them into computer, I can omit details I found trivial in the process and just keep only the essentials. This forces me to concentrate on the content, without worrying about style, such as deciding on the font size from the begining.)

There is a delicate balance between humanity and tools. Tools won’t steal the thunder of the content; instead, they assist in delivering and expressing it in a natural way, where the care and empathy of a human being are best presented. The uniqueness of content is embedded in the form, facilitated by new tools like AI. With those tools, the content could reach out to more people, maybe that is one message from universe so that harmony could be echoed and vibrated in a bigger scale.

On one side, there is the human; on the other, there are the tools. If tools like AI can lower barriers for knowledge sharing, it means the status of humans is actually elevated and levelled up, much like what happened in the Humanism movement.

It is the human, in the midst of a constant influx of information, who takes time to be willing to be bored, alone with his thoughts and feelings, and digs himself out on his own, that can truly make the best use of the tools and information without being drowned by them in our time. As what substance imparts infinite attributes to Spinoza, to get into that deep primordial state helps us to not be blinded by the diversities of forms on the surface, as it is the source and origin of all forms, hence to express it in the most natural way without being limited or overwhelmed by the tools and styles - make something wonderful.

One name just popped into my head as I’m writing this part. That is Jimi Hendrix - a combination of creativity and application of technology. The guy who pioneered the use of all the possibilities of the electric guitar as an instrument, an all-encompassing genius, shows us how to use technology to explore, broaden, and expand one’s creative potentials and take expression(guitar) of one’s life(music) into uncharted territories.

“I want my music to go inside the soul of a person. You know, for me it’s colours. I want people to feel the music the same way I see it. It’s just colours. That’s it. The rest is just painted with a little science fiction here and there”. (Jimi: All Is By My Side)

I wonder what Jimi Hendrix would say in this AI era. But I guess I know his answer. That unique and human in quality, the intuition, the raw imagination and direct feeling from life, to express one’s inner voice/serenity, where that limitless goes beyond limitation, is really what matters utmost. Protect that inner world, that purity.

Interestingly, there is a statement from an ancient Chinese art critics, Chang Huai’s Treatise on Chinese Painting :

Only he who reaches Reality can follow Nature’s spontaneity and be aware of the subtlety of things, and his mind will be absorbed by them. His brush will secretly be in harmony with movement and quiescence and all forms will issue forth. Appearances and substance are caught in one motion as the life breath reverberates through them. He who is ignorant of Reality becomes a slave of passion and his nature will be distorted by externalities. He sinks into confusion and is disturbed by thoughts of gain and loss. He is nothing more than a prisoner of brush and ink. How can he speak of genuine works of Heaven and Earth?

Summary

A crisis, on the other hand, could lead me to have doubts and evoke a multitude of questions about myself, serving as a great guide. I’ve found that if I don’t reject negative feelings or try to hold on to positive feelings, I’m in a better position to receive the subtle messages the universe. Like what Horace’s Epistle to his teacher Maecenas where he clamins not to be devoted to any particular sect but is rather an eclectic by nature (source: Nullius in verba wikipedia).

Nullius addictus iurare in verba magistri, – quo me cumque rapit tempestas, deferor hospes.

(“No one is addicted to swearing by the words of the teacher, – to which whenever the storm takes me, I am carried away as a guest.”)

I mean it’s always there, and it’s just that I rejected the inner child and closed off the inner light, either voluntarily or involuntarily, turning to seek validation from outside for a long time. I put up an invisible chain a shackle on myself. Time to put down.

In this stage, I feel that life is more about art and the subconscious than just being a logical/rational calculator and conscious(The Matrix). That spontaneous reflection, that harmony, that fusion of subjective and objective reality maybe mostly does not take place in the conscious realm through a logical process or any artificial effort. Still, it doesn’t need to be mystic, abstract, or spiritual; just something we can access in our ordinary daily routines. It is inside us. We’re all empowed to play, create, connect and heal.

Aren’t we all still children inside our hearts? Maybe the kid/child is our greatest teacher from universe. Wow, my friend, let us be reminded: we already have this greatest teacher inside us.

To play, and make living itself an art, as artist of life.

I want to end with quotes from Ernest Becker’s «The Denial of Death»：

Childlike foolishness is the calling of mature men. … [N]eed for legitimate foolishness.

[T]he only secure truth men have is that which they themselves create and dramatize; to live is to play at the meaning of life.

Quitting Coffee

2023-08-03T11:55:32+00:00

Why do I want to quit coffee?

I’ve noticed that my sleep quality hasn’t been great lately. I feel like coffee might be contributing to that.

I drink pretty much every day, one cup of instant coffee in the morning and another one at noon for maybe 10-12 years. Even though money is not that much, it adds up, it’s not a small amount.

But lately, I’ve realized that every time I drink coffee, it doesn’t give me the same feeling of freshness it used to. It’s like I’ve lost that initial excitement about it. Sometimes, it feels no different than drinking water, and it’s not even making me more alert or focused anymore. So, what’s the point of continuing to drink it?

I decided to set a goal for myself: to change my coffee-drinking habit and eventually quit it altogether. I know it might be tough to completely stop right away, so I thought of starting with a short-term goal: going from having it every day to just once a week. That way, it doesn’t feel too overwhelming, and I can gradually reduce my coffee intake.

Quitting Journey

At the beginning, I started drinking coffee for the purpose of staying alert, as caffeine can accelerate heart rate and bring a sense of pleasure, making the brain more awake. However, when I began the first day of quitting coffee, I experienced feelings completely opposite to my initial encounter: it wasn’t pain, anxiety and depression, but rather boredom and emptiness. I couldn’t lift my spirits, and my brain was kinda foggy. I felt like a lifeless body, completely unable to concentrate. I looked up some information online and found out that these were typical addiction withdrawal symptoms during the early stages of quitting.

After the second and third day, the discomfort had slightly eased, and the concentration got improved a bit. However, I almost couldn’t resist the urge to drink coffee, so I opted for a cup of tea instead, which helped to alleviate the itchiness. Finally, the weekend arrived, and I couldn’t wait any longer; I had a cup of coffee.That taste indeed brought back that freshness of the first time I drank coffee, and after finishing it, I could feel my heartbeat accelerate, my body and mind become invigorated.

In the following second and third weeks, the quitting process became relatively smooth. The craving for coffee was no longer as strong, and my attention gradually returned to the level before quitting. Overall, I started feeling more comfortable and at ease.

habit tracking calendar

When I began my coffee quitting journey, following online recommendations, I bought a calendar paper of habit tracker. If I successfully abstained from drinking coffee on a given day, I would mark a check on it with a pen. This simple ritual made me feel satisfied because each check represented accomplishing a small goal. (one step closer)

As time went by, about one or two months later, I no longer felt the need to drink coffee purely for its stimulating effects. However, I still maintained the habit of drinking it once a week, as completely giving it up would take away some pleasures. But now, I don’t stick to a specific time for drinking; instead, I randomly choose a day within the week to drink. Additionally, I’ve been trying different styles of coffee such as latte, Americano, cappuccino, hot, and iced, to bring some novelty. Well, I could still enjoy the freshness of coffee without indulging.

Reflections on Quitting

During this process, I noticed the addictive aspect of coffee. It started with just having it occasionally, but slowly, I found myself needing it every day. As I continued with the same amount, the stimulating effects and pleasure gradually diminished. Drinking coffee just turned into a habitual behavior, and I felt subconsciously compelled to drink it. Not having it made me feel like something was missing, but drinking it didn’t provide the same effect as before, and I even felt less alert. To keep up that effect as before, I need add up the amount of coffe intake. If I skipped one day, I would feel extremely low, bored, and a sense of emptiness, which would drive me to seek more coffee to alleviate those negative feelings.

This addiction felt similar to the gaming addiction I experienced during my student days, as well as the addictions some relatives and friends had to somking and alcohol. I became curious about how coffee affects the brain and body, so I looked into the mechanisms of addiction, and one word stood out: dopamine.

Dopamine is known as the feel-good neurotransmitter—a chemical that ferries information between neurons. The brain releases it when we eat food that we crave or while we have sex - all those behaviors would lead to a higher chance of survior(complete to get more resources and higher chance to reproduce etc ), contributing to feelings of pleasure and satisfaction as part of the reward system. This important neurochemical boosts mood, motivation, and attention, and helps regulate movement, learning, and emotional responses.

https://www.psychologytoday.com/us/basics/dopamine

High dopamine levels in the brain are associated with feelings of excitement, motivation, and even euphoria. On the other hand, low dopamine levels can lead to feelings of low mood, depressed.

(A simple description is that dopamine fluctuates around its baseline level, oscillating between peak and valley values, much like a sine wave.)

Dopamine plays a crucial role in the reward system of our brain. As humans, our brains are hard-wired to seek out behaviors that release dopamine in our reward system. Dopamine is released when the brain is expecting a reward.

Just imagine the first time you eat some delicious food at a restaurant or experience other pleasurable stimuli like video games or intimacy. The moment you taste the food or engage in the activity, a flood of dopamine is released in your brain. It makes you feel high and possibly euphoric, which reinforces the desire to repeat that experience with the same stimulus - a cycle of motivation, reward, and reinforcement. So, while dopamine itself doesn’t directly produce pleasure, it does strengthen the feelings of pleasure and reward associated with specific behaviors or substances.

Next time, when you think of the food you ate last time, you already start to feel high because of dopamine release, even with just mere anticipation.

Unexpected rewards increase the activity of dopamine neurons, acting as positive feedback signals for the brain regions associated with the preceding behavior. As learning takes place, the timing of activity will shift until it occurs upon the cue alone, with the expected reward having no additional effect. And should the expected reward not be received, dopamine activity drops, sending a negative feedback signal to the relevant parts of the brain, weakening the positive association. Source： Dopamine, Smartphones & You: A battle for your time

When dopamine is released, it needs to activate dopamine receptors to generate an excitatory response in the brain. Once the receptor gets activated, dopamine gets recycled back to our body with the help of transporters. The dopaminergic system maintains a delicate and subtly dynamic balance. It is like a seesaw; when one end (pleasure) goes up, the other (pain) goes down. However, it slowly comes back to normal, just taking some time. The body will rebalance, and the baseline level will return to normal. That’s the natural state of the dopaminergic system - dopamine is generated by the body and recycled by the body.

However, in this digital era, the stimuli have increased exponentially compared to what our ancestors experienced not that long ago, around 100 years ago. We now have access to TV, computers, and phones, and the internet has become dominant worldwide. Additionally, there are drugs and highly processed chemicals, such as sugars, that cannot be obtained from nature simply by chewing on coca plant leaves like Andean peoples did. These chemicals are highly purified and isolated from plants. They deceive and manipulate the dopaminergic system by either delaying recycling, prompting the body to release more dopamine, or increasing the level of activation on dopamine receptors.

Those are totally unseen before. Like Nietzsche says, ‘God is dead.’ Ever since then, those natural limits forced upon us to keep us in well-being are gone. The dopamine system was welcomed to a Brave New World. Now, you have so many choices, and of course, we’d like to give a try to Amusing Ourselves to Death. But why?

Those chemical stimuli lead to a very high dose of dopamine release (the peak is higher) compared to natural stimuli, so much so that anyone who tries those stimuli would immediately lose interest in the natural ones our ancestors had. However, you won’t stay that happy forever; even though external chemicals can easily overpower the native ‘inferior’ and ‘inefficient’ primitive dopamine system, much like an elephant stamping on ants. The balance is broken, and the primitive dopamine system fights back by reducing the number of receptors or, in extreme cases, sacrificing some cells or organs to protect itself and try to regain balance. These protections could possibly cause irreversible damage to the body. So, your dopamine baseline level also gets lifted up along with your peak dopamine levels.

Again, remember that the pleasure you feel is equal to the Peak - Baseline level. It makes sense now that I take the same amount of coffee every day, but the pleasure I get is weaker and weaker, just like the law of diminishing returns in economics.

source: How do drugs affect the brain?

So, they need to get more chemical stimulus to maintain the same level of pleasure they feel. Quite ironically, when dopamine peaks for some time, the dopamine baseline also adjusts itself to a higher level.

Now, think about what happens if you suddenly remove the stimulus? peak - baseline level will give you a negative number, leading to pain, boredom, and depression, along with physical suffering. Since the baseline level has become so high and they are used to high dopamine stimuli, they just can’t stay satisfied with natural stimuli. Even eating delicious food would hold zero interest for them; the little enjoyment from that is barely recognizable. At this low dopamine level, time passes more slowly in psychology, making the pain even more pronounced, leading to negative reinforcement (Davis double-killing effect – does anyone remember the death spiral of the stable coin Luna?). It is easy to understand why they could fall back and relapse as it is the most convenient way to get rid of that uncomfortable feeling.”

source:杜新忠戒毒网

Smartphone Addiction

My daily commute routine goes like this: In the morning, after waking up, I first check my phone. While riding my bike to the subway station on my way to work, I play videos in the background from platforms like Bilibili or YouTube to listen to.

During the subway ride, I use the time to learn Duolingo in 5-minute units to improve my Spanish. After that, I watch short videos on various platforms like Tiktok, Bilibili, and YouTube Shorts. Once I arrive at the office, I turn on my laptop and do some warm-up activities, first checking V2ex and YouTube, before getting into work mode. Throughout the workday, I occasionally check my phone, including during restroom breaks. Sometimes, I do the same during lunch with my colleagues. On the subway ride home, feeling tired, I take out my phone and watch short videos to learn about carp fishing.

Yeah, I pretty much stick my face on the phone.

The problem is that I’ve started feeling like I’m caught in a negative cycle. I feel less energized, find it hard to focus, and have less patience when my two and half years old son asks me to play with him. One weekend, when I took him out to the kids’ area of a mall, I looked around and realized that most parents were just playing with their phones - the majority of them using short video apps.

This feels like a black hole just sucking in most parts of my life. I feel lost and trapped.

It turns out, for instance, TikTok and those super-short clips are like heroin for our dopamine system, just like the above-mentioned chemical stimuli. TikTok’s algorithm is specifically tailored to keep our dopamine levels at a concerning high. The random, intermittent, unexpected rewarding path tricks our brain, much like hitting a jackpot, causing it to release dopamine. We become lost in endless, mindless scrolling because our brains crave new, random videos to get a new shot of dopamine.

Andrew Huberman nailed it:

Addiction is a progressive narrowing of the things that bring you pleasure. A good life is the progressive expansion of the things that bring you pleasure”.

DOPAMINE DETOX: Take Back CONTROL Of Your Life & STOP LAZINESS! | Dr. Andrew Huberman - YouTube

Bertrand Russell once said well:

“The secret of happiness is this: let your interests be as wide as possible, and let your reactions to the things and persons that interest you be as far as possible friendly rather than hostile.”

don’t think I’m friendly to those stimuli that interest me. Certainly, I won’t ask them for permission; I just take them commandingly :) Well, Russell probably didn’t know that things and persons that interest you are probably the devil pretending like a nice guy.

Comparing to learning or working, immediate-gratification activities could easily get us hooked. That’s why it is so hard for me to get into a working state. The beforehead immediate-gratification activities raise my dopamine baseline level, but at the same time, they exhaust me mentally and physically. Once the stimulus gets pulled out, the dopamine level drops, and the symptoms of dopamine depletion kick in.

Focus is hard to maintain, and I feel bored in front of low-and-long-dopamine-reward tasks. So it is better to arrange tasks in a dopamine-scientific way. (By the way, has anyone really done serious learning on TikTok? I did try to learn carp fishing, but after a while, it starts recommending videos of girls shaking their asses? WTF, I neither searched for that type of content nor accidentally clicked on one @_@)

Another thing is, just don’t stack dopamine activities. Avoid multitasking on high dopamine activities. Do them one by one as much as you can or engage in multiple tasks but keep it as natural as possible. When you eat, just eat; don’t eat while playing with your phone at the same time unless you have to. You might think you can kill two birds with one stone, but in reality, you might kill two stones with one bird. Essentially, you ruin two pleasurable things at the same time. When tired, sleep. When hungry, eat.

Someone asked a Zen Master, “How do you practice Zen?”

The master said, “When you are hungry, eat; when you are tired, sleep.”

“Isn’t that what everyone does anyway?”

The master replied, “No, No. Most people entertains a thousand desires when they eat and scheme over a thousand plans when they sleep.”

I’m so fed up with this rat race. After I come back from work, I try to put everything down and empty my mind, and play with my son. This time, during my commute back from the office, I don’t play with my phone. Instead, I observe and look around at people and what they’re doing. I even close my eyes for a few minutes just to listen to the sounds of the train. That’s it.

When I play with him, I notice his expressions, reactions, and we work together to build something with Lego blocks. Sometimes, he is not patient and will tear the building down immediately. Other times, he surprises me with actions that are out of the box from an adult’s thinking in my opinion. It’s so fun to watch. And maybe just minutes later, he would start playing on his own and seems not very interested in having you around.

It is definitely a refreshing experience, and I feel a deep connection between us. I wasn’t eager to seek high dopamine release, but I did feel a sense of peace and enjoyment in that moment of being fully present.

What I have decided to try in the following weeks:

When you eat, just eat.
When you defecate, just defecate.
For one random day of the week, drink only water - no tea, no coffee.
Delete TikTok
Spend half of the time during the week not listening to anything on your commute.
Half of the time on the metro, try take off my glasses (I’m nearsighted) and simply observe the surroundings with everything blurred.

Dopamine traps in building habits

1. Visualize too much in the starting time of a new habit.

When we start building a new habit, we tend to think or visualize how great we would become after adopting the habit. Whether it’s losing weight, quitting smoking, or taking up running, we can’t help but imagine and fancy a better version of ourselves after a certain period of time. “Oh, I will lose 20kg in 3 months” or “I will run 10km in 2 months,” we think optimistically. This anticipation triggers our brain to release a high dose of dopamine, driving us to pursue our goals. However, this heightened dopamine level can also lead to risky decision-making, causing us to seek maximum rewards as quickly as possible, which might result in compulsive behaviors, gambling for instance.

The consequence of this high dopamine level can be setting inappropriate goals for each step, overestimating what can be achieved in the short term. While the initial motivation might be high, it won’t last long if those goals cannot be realized in the real world. The actual rewards often turn out to be not that good than our anticipations, making it challenging to maintain our momentum and enthusiasm for the new habit.

You need to lower your expectations upfront. After cooling for one week, then start habit planning. For instance, when I started learning Spanish on Duolingo, I set the goal for 5 minutes, the minimum of 10, 20, or 30 minutes options. Even though the first time I finished with ease and felt a strong desire to do more units, I kept it to just one for the first day. This is because doing more upfront can burn out my motivation in the long run.

High expectations can lead you to believe you’re on top of the world, but it’s better to remain humble and stick to yourself. Better stay humble like a fool.

2. Perfectionism in pursuing streaks.

In Duolingo, every day you finish a goal, you are rewarded with a nice and attractive animation to show that you win. It gives you a pleasing effect. Additionally, you can see the streaks on your calendar. Observe the line that crosses all the days on your calendar, that smooth line. It makes you feel good and increases the likelihood that you will continue doing it tomorrow.

Take a look a the check animation:

And habit track calendar:

I have a streak of 998 days pretty like 1000 days at this time of writing. I can’t imagine what would be it if I set my goal to 10 minutes or 30 minutes a day(I’m not sure they have so many units then).

I had a perfect streak, but there were a few times when I almost missed one or two days. One day, I thought I had finished a unit, but around 11:35 PM, near midnight, it sent a notification that I was going to miss the streak. Then I realized I had actually forgotten to do it. So, I rushed to open the app and finish one unit.

The problem I have noticed over time is that this beautiful streak and the check animation, which were supposed to provide motivation and pleasure to keep me going, now put some cognitive burden on me. I’m more concerned about not breaking this perfect streak and seeing the nice animation, rather than actually learning from each unit. In other words, I have become addicted to the habit-building auxiliary tools in this process. The real purpose, the original vision, is gradually fading away amidst these daily autopilot routines, pursued only for the sake of formalities.

The more creative and dopamine-inducing the animations and other tools become, the more likely you are to put yourself in these shackles. The pleasure and dopamine you get from actually doing the task are probably less than what you experience from the goal-check and streak. This whole thing goes completely against its own purpose, doesn’t it? It will backfire!

Uncertainty

As I grow older, I realize that uncertainty is probably the submerged part of the iceberg, and I understand the need to shift from a certainty mindset to an uncertainty mindset. Life throws you curveballs, and you don’t really have as much control over your life as you may think. Things don’t work in a linear way; there are ups and downs, and that’s what makes it fascinating. You have limited control over how low the downs can be.

It’s very likely that you’ll miss a step. It’s a sure thing. I know I have been keeping the streak for almost 1000 days, but I don’t know what time, but someday this streak will definitely break. I caught COVID twice, and there are days when I just don’t have the mood or energy to do things, leaving me with brain fog all day and lacking the motivation to continue.

And that slip-up will break the perfection of the streak, as if you have lost all the progress. It is a nail stuck inside your brain, causing so much distress that you feel compelled to start all over again just to get rid of those negative feelings and emotions arising from an imperfect streak formality.

It’s so harsh that it often leads people down, even causing them to question themselves. “You just can’t do anything! I told you,” a voice from the depths of their mind echoes, paying tribute to the «Notes from Underground».

The downward spiral is as destructive as Luna’s crashing. The depletion of dopamine levels you build over time can lead to feelings of pain, boredom, and even depression. In some extreme cases, it may result in a ferocious attack against oneself, as attacking yourself seems like a quick way to find relief.

Sure, you can go back to the previous day and mark a check to make it look like a perfect streak. But my friend, you can’t lie to yourself. This is the worst thing I could imagine you doing – cheating on your inner self for the sake of external satisfaction. This will only accelerate the process for you to give up that habit, and even worse, you will have done serious damage to yourself.

“You’re not only a quitter, but also a liar — not just to someone else, but to yourself. You deserve every bad thing you’ve had in your life,” a voice inside your head whispers.

Certainty mindset definitely plays a positive role and helps us progress, but it can also become an obstacle. To truly explore ourselves, we need to constantly peel off the layers of fixed pattern and habitual thinking. We should avoid framing ourselves in a confound to seek excessive security and comfort. Let’s not attach ourselves to specific styles, like streaks, but instead, focus on the essence of our actions and eventually on our own being.

Lastly, let’s be honest with ourselves, expressing our thoughts fully, being present and wholehearted in the process.

Perfectionism is the number one enemy in building long-term habits. There is nothing wrong with pursuing the perfection of styles and formalities, but let’s not forget the core and fundamentals. Otherwise, as Bruce Lee said in :

It’s like a finger pointing away to the moon. Don’t concentrate on the finger or you will miss all that heavenly glory.

Perfectionism is a fragile and ephemeral thing in the face of the force of uncertainty. In Taoism, the concepts of perfect and imperfect do not exist in isolation; much like yin yang, they are indivisible. One cannot selectively choose perfection and expect it to remain in a perpetual flawless state. It is futile to become attached and cling to it.

Feel and Being

We have a big picture of how the dopamine system and brain work. When dopamine levels reach their peaks, don’t intentionally introduce more unless naturally. And when dopamine levels crash, also avoid intentionally seeking stimuli to get rid of those uncomfortable feelings. The key is, how do you know it? How are you aware of that you’re in that kind of state? That’s the number one question.

This may look easy, but it is actually very hard to do. Knowing in a dopamine baselevel is quite different from knowing in a high or low dopamine level. The standard way of saying things may not apply to each individual because each one is different. Therefore, there is no standard and concrete answer to this. However, we could start by deliberately building our awareness.

Building awareness requires openness, where one must fully acknowledge their ignorance and empty their cup. This will naturally lead to curiosity about oneself, its authentic existing in this cosmos. Become aware of your body, emotions, thoughts, and surroundings.

Number two: How can we deal with that state and those feelings? Awareness is kind of like letting it go, observing from a distance. Feel the body, the emotions, and all those wandering thoughts before instinctively reacting to them. Experience the moment, be fully present. Those boredom and pain are interesting to watch, aren’t they? It’s just like when we go outside to a park, sometimes my son stops on the trail and watches the ants moving here and there with focus, curiosity, and patience showing in his face.

We have lost that subtle sensitivity for quite a long time: sensitivity of our bodies, sensitivity to our surroundings. The illusive feelings of “secure” and “comfortable” that we build over years of habitual activities lead us to believe we hold ultimate authority, until life throws another curveball. We are becoming rigid, more like rats in a race, losing all those colorful, dynamic, vibraint and glowing individual temperaments that every authentic, honest, and unique human being possesses from birth. The light inside is off, leaving us in complete darkness.

Be open to discomfort, be curious, resist the immediate urge to place judgments, and feel and experience it as if it were a precious gift.

Carl Jung:

Psychological or spiritual development always requires a greater capacity for anxiety and ambiguity.

Then discomfort (pain/boredom, just like dopamine depletion) can be a great hint and clue to break free from our rigid thinking developed through our daily routines. We are no longer on autopilot; it’s time to step back and check, which is actually a good thing. As Bruce Lee nicely put it in 1970 when he hurt his back very badly, which could have possibly led to paralysis, in his letter to his friend:

“I mean who has the most insecure job than I have? What do I live on? My faith in my ability that I’ll make it. Sure my back injury screwed me up good for a year but with every adversity comes a blessing because a shock acts as a reminder to oneself that we must not get stale in routine. With adversity you are shocked to higher levels if you allow yourself to go beyond your current circumstances.”

”Remember,my friend,it’s not what happens that counts;it is how you react.Your mental attitude determines what you make of it,either a stepping stone or a stumbling block.“

source: Bruce Lee Podcst: #11 Walk On

He goes on to say:

… to understand your fear is the beginning of really seeing. Fear compels us to cling to tranditions and gurus. There can be no intiative if one has fear. The enemy of development is pain phobia - the unwillingness to do a tiny bit of suffering. As you feel unpleasant, you interrupt the continuum of awareness and you become phobic.

Experience the pain and pleasure as it is, but not to attach to it or try with any thoughts to blockade or avoid it. It is in this full totality, wholeness of immersive experience, we could given up all those impluses to control it, impose our will on it, hence in a natural and Wu-wei(The Art of Not Forcing) way. If my son is 2 years and half old, then I’m the new born baby to him, not as an adult. I’m learning to search myself as he is learning to grow. Maybe there are some teachings I could provide for on how this external world is running, hey, other than that, all I could do is just provide some guidences for him in his unfolding journey. Clearly it will expose me to those anexity and uncertainity of refusing to impose some social norms or conventions or dogmas from outside on my son just for sake of my comfortness and sense of security and certainty. As Carl Jung said: “Knowing your own darkness is the best method for dealing with the darknesses of other people.” , and Confucius said: “You should treat others the way you like to be treated.“, and as I slightly modified saying: “Don’t do unto others what you don’t want done unto you PLUS what you want done unto you too.” , I don’t want to possess that easily-and-quickly-gained relief and temporary good feeling by follow those social norms or conventions without doing my homework to examine what those really are, without feeling my son’s feelings in his shoes. Of course, I dont’ want my son to conform to my authority and not be happy.

Sympathy and compassion must be granted by a total self-rupture to break those rigid patterns, layers, armours and so called “I think it is good for you” kinda authority, so we could get close to our soul in order to fully feel other’s soul in a equally way and be able to listen and echo back.

Therefore I must prepare myself in a fully peaceful state to embrace those discomforts and uncertainities. Not only it is always me, as a father, who takes the responsibility readily anf full-heartly to protect and give my kid some room of purity to let his nature blossom, but also it let me to have chance to search myself better. How about expericing and learning together in this life journey? I’m voluntarily and effortlessly putting on those shackles which seems to give up some freedom I have, but I know I’m getting a total and full freedom from the soul. To some extent, I’m, from now on, a Free Man.

German mystical Meister Eckhart once said:

Truly, it is in darkness that one finds the light, so when we are in sorrow, then this light is nearest of all to us.

There also reminds me of one time when my son and I were walking in the countryside. He looked at something strange and seemed pretty scared. When I inched closer, I realized it was a long piece of snake skin! My son said it was a snake. I reassured him, “It is not a snake, don’t worry. It’s just the skin that the snake shed.” He looked puzzled, and a question popped into my head: “Why do snakes shed their skins?” It turns out snakes shed their skin to allow for further growth and to remove parasites that may have attached to their old skin. At this time, it is probably the vulnerable time for the snake. They shed their skins every two or four times a year. That just gives me an aha moment – we need to constantly peel off our old skin and grow the new skin, and this requires a greate capacity of anxiety, bordem, randomness, uncertainty, fear, even pain. Just keep re-centering yourself, like water, no staleness and no rigidness, flow as it is. The process of snake rubbing against a rough, hard object to peel off his skin is kinda similar to our grindings in those situations of setbacks and low moments. Maybe we can use those moments as a chance to shed our old selves and be reborn anew.

“Jeet Kune Do teaches us not to look back after the course has been decided. He treats life and death indiscriminately. “ (…) To express yourself freely, you must forget yesterday. From the “old” you get security. “New”, you gain fluidity.” — Bruce Lee.

I did learn fishing from Chinese Tiktok and that skills were pretty good. Those are some beautfiy little fishes with nice colors and scales textures. My son got really interested on those fishes.

A losing streak could server as a reminder for us to examine our actions to have self-reflection and improvement: Have we deviated from the essence? What efforts should I focus on? Should I consider making slight changes in direction? It helps us to reassess our goals, strategies, and priorities, making adjustments if necessary. Embracing the lessons from losing streaks can lead to growth and progress on our journey.

There is another question merged from this: Doing is imporant, and you need take action, but is it sufficient? I vaguely recall Martin Luther when in The Reformation, in this book «A Treatise on Good Works» , he explains that faith are not earned by good works from outside, but that good works flow from faith from inside. You dont’ do the good work just for the sake of the reward from God or the fear of Law. You do the good work just because it is naturally who you are by your faith. And in this «A Preface to Romans» he saids:

The person with a true faith will perform the works of the law because the works of the law will be what they desire to do; the works are thus an outworking and a result of the faith, in the same way that heat is an outworking and a result of the fire.

To quote from Lao Tzu:

The man of superior virtue is not [conscious of] his virtue, And in this way he really possesses virtue. The man of inferior virtue never loses [sight of] his virtue, And in this way he loses his virtue. 上德不德，是以有德；下德不失德，是以无德。

Instead of obsessing over what you need to be, focus on what you are - the Being. Doing becomes a natural extension of being; it overflows and radiates from your authentic self, akin to the concept of “The One” in Plotinus’ philosophy. Being is not in the future, nor is it in the past, instead Being is in this moment, present.

Take a step back to reflect, contemplate, as I’m growing older, is to constantly search the true self and acheive the self-actualization through the process. What you are is something only you could dig it out, no one else could do it for you. Once you get some glimpse of it, you kinda know who you are. Then try stay quiet for some time, let it emerges and unfolds, don’t be an utilitarian for moment, or you won’t heard it from the background. The thing is that it just takes no effort, put other way, those actions or habit building is in reality effortless. Being present in those actions is a way to experincing with inner oneself. No external or extra drive or movitation is really needed or placed to let you “make effort” consciously. Shifting the focus from outcome to process then is a spontaneous thing which the inner force will empower us in a tender, sustatainable, healthy manner in long term.

Yeah, building habits is a great way to gain a better understanding into oneself. Knowing oneself is a dynamic process, and it is undoubtedly challenging. Doing a deep dive into oneself’s inner world is tough and even daunting. In a world where “God is dead,” you become the light to yourself. You may feel alone at times, but you shouldn’t feel lonely — There are and there were so many comrades and the deep connection across time and space between human beings bonds us together. Walk On!

Know the principle, follow the principle, dissolve the principle, and forget the principle.

Chop Wood, Carry Water!

Dont’ think, Feeeeeeeeeeeeeeeeel, my friend!

戒断咖啡

2023-07-28T04:55:32+00:00

为什么要戒断咖啡

感觉睡眠质量比较差
一个月咖啡钱也不少

目前：早上一杯挂耳，中午一杯咖啡，得有16-18元每天。几乎每天都喝，这个习惯已经有很长时间了，至少5-6年起步了。

其次：每次喝咖啡感觉并没有当初新鲜感觉，对于我而言，咖啡失去了其初心，甚至跟喝水没什么区别，更别提能够提神于是你不的想何必在喝了？

于是我给自己设定了一个目标：改变喝咖啡的习惯，戒掉咖啡。显然一开始要求彻底不喝咖啡很难，我觉得可以设置一个短期目标：从一天一次改为一周一次。

戒断历程

一开始喝咖啡确实也是为了提神，因为咖啡因可以加快心率并带来一种愉悦感，让大脑变得清醒。然而，当我开始戒断咖啡的第一天时，我感受到了与初次接触时截然相反的感觉：并不是痛苦和烦躁易怒，而是无聊、空虚，提不起精神，头脑昏昏沉沉，感觉自己像个行尸走肉，完全无法集中注意力。在网上查了一些信息，发现这是典型的戒断初期症状，有建议尝试转移注意力等方法，但我觉得效果一般。

经过第二天和第三天，难受的感觉稍微减轻了一些，注意力也稍微能够集中。但我仍然忍不住想喝咖啡，于是我选择了喝了一杯茶，这样就没有那么心痒难耐了。终于等到了周末，我迫不及待地喝了一杯咖啡，那杯咖啡让我感觉它找回了它的初心。味道确实让我回忆起第一次喝咖啡的感觉，并且喝完之后能感受到心跳的加快，身体和精神也振奋起来，思维变得更加活跃。

随后的第二和第三周里，戒断过程变得相对顺利。对于咖啡的渴望也不再那么强烈，注意力也逐渐恢复到戒断之前的水平，整体感觉变得越来越舒适和自在。

类似这种daily planner 日历计划表

准备戒咖啡任务开始时，按照网上的推荐，我买了一个日历纸。如果我当天成功戒断没有喝咖啡，我会用笔郑重其事地在上面打一个勾。这个简单的仪式让我感到满足，因为每次打勾都代表着完成了一个小目标。

随着时间的推移，大约过了一两个月，我基本上不会因为咖啡因的刺激效果而去喝咖啡了。不过，我还是保持每周喝一次的习惯，毕竟完全放弃也是让生活失去一些乐趣。但现在我选择不固定某个特定的时间，而是随机地在一周中的某天喝一次。而且我尝试着选择不同口味的咖啡，如拿铁、美式、卡布奇诺等，冷的热的，这样能带来新鲜感。基本上每次喝咖啡的时候的感受基本还有像第一次喝那样那样的强度，但是了，自己也不会想着为了提神而需要喝更多的咖啡。

戒断的思考

庄子：“丧己于物，失性于俗，谓之倒置之民。”

在这个过程中，我注意到首先是咖啡上瘾这个环节。一开始可能只是几天喝一次，但慢慢地，我渐渐需要每天喝。后来，早上一个挂耳，中午一杯咖啡成了必须。然而当我保持相同的量时，提神效果和愉悦感却逐渐减弱。咖啡渐渐变成了一种日常化的习惯行为，潜意识里觉得必须喝，不喝会感觉缺少了些什么，但是喝了却感觉效果不如以前，甚至困意不减。

当我增加咖啡的饮用量和浓度时，那一刻感觉就好像找回了初次品尝咖啡的感受，但是过几天，这种初次的体验逐渐下降，最终又回到了那种平淡无奇的体验。以至于，如果有一天我不喝早上的挂耳咖啡，我会觉得非常难受。我开始意识到我对咖啡因产生了依赖，我觉得我已经对咖啡上瘾了。

这种上瘾跟我学生时代的游戏瘾、旁边亲戚朋友烟草和酒精的瘾感觉是相似的，我很好奇其咖啡如何影响大脑和身体的，于是查了下上瘾机制，其中离不开一个词：多巴胺。

多巴胺是一种能带来能量和动力的神经传导物质，是愉悦的操控者，让你做事的动力和积极性（奖励、愉悦或满足），也是让你产生抑郁。，多巴胺对我们的身心健康有着至关重要的作用，同时还跟愉悦和满足感有关，当我们经历新鲜、刺激或具有挑战性的事情时，大脑中就会分泌多巴胺。它与人的情欲、感觉有关，它传递兴奋及开心的信息，也各种上瘾行为有关。如果人缺少多巴胺的受体，就会抑制兴奋。增加多巴胺，就能让人兴奋，心情愉悦，但是它会令人上瘾。

多巴胺是带来能量和动力神经递质，大脑中的多巴胺经过突触将信号传送到其它神经细胞，其中有一条涉及“奖赏系统（reward system）”的路径被认为与多巴胺传递快乐信息密切相关。所谓奖赏系统，实际是一组神经结构，旨在维护动机显著性（动机、需求、喜好等）、联想学习和正面情感（尤其是以愉悦感为核心的情感）。换言之，动物和人的中枢神经系统具有奖赏机制来加强和激励对机体有益的行为，以利个体生存和种族繁衍。更深入的研究表明，当大脑发现获得奖励的机会时，它就会释放出多巴胺，大量的多巴胺并不能直接产生快乐感，它更像是一种激励，让我们发现如何才能得到快乐，而且愿意为了获得这种感觉付出努力。简言之，多巴胺的效用是期待奖赏，而不是获得奖赏。

就像你看到食物的照片或者网上某个裸露的女人照片，你的大脑中那个原始的不受控制的奖励系统就会开启，多巴胺神经元会被激活，从而产生强烈的多巴胺信号，这会引起愉悦的感觉，去做出这种行为，哪怕这个甚至不是实际吃得到摸得着的实物，这种期待奖赏的快感会促进我们对这种行为的重复。在原始人类时代，在危险遍布的环境里，你是需要足够的动力和奖赏（释放多巴胺）去捕猎来获取食物或者交配到更多的异性来获取个体和种族的延续，这时候快乐愉悦是驱使人类发展和更好适应环境的渴望和动力。当多巴胺水平下降时候，你就会失去对于奖赏的期待，情绪低落，也就没啥动力，奖赏没啥诱惑了，直接躺平开摆，进入贤者时间。

(简单的一个描述多巴胺基准水平在上下在顶峰和谷底值之间波动，有点点类似正弦波)

多巴胺作为神经递质或激素，需要作用于多巴胺受体才能产生兴奋效应，也就是与突触后膜上的受体结合。如果受体已经被其他物质绑定，那么更多的多巴胺分泌和浓度也不会产生对应的效果和作用。受体的数量和处理能力在生理上有一个最大值和最小值的限制，就像人耳对声音的感知范围有限一样。

过高的多巴胺浓度就像刺耳的声音，会让人不适甚至造成危害。它可能导致血压升高、情绪失控，并在长时间高强度刺激下对细胞产生不可逆的损伤。相反，过低的多巴胺浓度则容易导致情绪低落、记忆力降低，甚至陷入抑郁。

身体本身拥有一套多巴胺的自我调节动态平衡系统，以维持适宜的多巴胺水平，也就是基准线。

同样当大脑接触到意料预期之外的奖励时（比如偶然打开了抖音）会增加多巴胺神经细胞的活跃程度，正面强化大脑中跟这个行为相关的区域。大脑开始学习并强化这部门神经回路。当下一次你想到抖音时（cue)提示时，奖励会提前，也就是预期到了玩抖音的快感，提前释放多巴胺。实际当你打开抖音玩时实现期待奖赏时，却没有多巴胺的释放了，因为这个在预期时候就释放了，没有多余的效果。如果预期中奖赏没达到和实现，比如抖音APP老是奔溃，那么负面反馈会给到大脑，弱化之前的正面反馈，你下次就没有那么强烈的欲望去重复这个行为。来源： Dopamine, Smartphones & You: A battle for your time

下面是不同活动对应的多巴胺释放的量的对比：

从上图看出来哪怕是毒品带来的快感的持续时间也是短暂的。快乐是短暂的，极大的快乐有极大的痛苦。

在没有外源性的多巴胺输入或者引起多巴胺系统规则的改变，食物和性等先天性的天然奖赏带来的多巴胺浓度是适量的，身体是可以花一定时间可以慢慢回收准备再次利用而恢复平衡，快乐和痛苦某种程度上说是一定天花板的。要提高快感和效果，那么需要从外部引入一些因素来欺骗大脑来获得药物奖赏：

增加浓度- 诱导多巴胺神经元释放更多多巴胺神经递质，一些物质如吸烟尼古丁和冰毒可以实现这个效果。
增加敏感度 - 通过比多巴胺更亲和多巴胺受体，提高受体的敏感度，某些毒品可以实现这个目标。
延缓回收 - 通过延缓多巴胺的回收，抑制转运酶DAT，使体内多巴胺水平能维持短期的高位，咖啡因就具有这样的作用。

来源:Yale Medicine: How an Addicted Brain Works

是不是多巴胺可以无限上升从而达到一致持续无限的快感了？为什么我每天喝一杯咖啡，但是体验到愉悦觉却是越来越弱？因为短期的大量的多巴胺释放会让持续刺激大脑突触结构上的受体，引起持续性的兴奋，这个异常的水平会让大脑做出适应性的动态调整和改变来尝试平衡。它会试图少突触后膜上多巴胺受体的数量来降低兴奋的作用，或者增加多巴胺转体的数量，加快多巴胺的回收。这些都会使得大脑对于刺激物不那么敏感，也就是感受到的愉悦度是下降的，当然同时也会降低对于天然奖赏的反应。这意味着必须加大使用剂量，才能维持和获得先前一样的快感，这个阶段叫耐受，跟胰岛素抵抗，抗生素耐药性有这相似的道理，经济学上管叫边际效用递减。

来源:杜新忠戒毒网

同样的也可以来解释为什么戒断时会有那么难受。上瘾之后，脑回路会形成了这块刺激-行动-奖赏链路时，大脑会记住那份快乐，在下一次继续寻求同样的刺激，但是此时如果已经没有外源物补充，没有进行高多巴胺活动，大脑就会产生戒断一些反应，比如焦虑，无聊，低落，无法集中注意力等等表现，有些甚至是痛苦的，低多巴胺那些自然奖赏已经是没啥动力了去做，为了减轻这些情绪带来的生理不适，不得不再次投入高多巴胺的活动，周而复始。

来源: How do drugs affect the brain?

这个戒断过程，对于像我只是解除咖啡因的而言，难受还是可以忍受的（身体反应不大），但是对于毒品上瘾者而言，大脑结构甚至已经发生不可逆的损害，那这个反应就是非常严重的了，在身体和心理都是双重的折磨，甚至是痛苦。痛苦这么大，以至于除了复吸，就只剩下死亡。极大的快乐意味着极大的痛苦，貌似从一个极端从钟摆一样到另一个极端，跟金融周期的繁荣和萧条周期一样。此刻心理时间上来说，快乐看起来是短暂的，而痛苦却要显得那么长久：快乐的时光总是那么短暂，痛苦、抑郁、焦虑、苦闷的时光总是停滞了一般，度日如年，牛短熊长。

短视频

我有病，我是一个病人……我是一个陷入短视频病症的34岁的中年人（尽管承认自己有病可能让人感到难堪，但对于一个主张坦然认识和跟随本性、本色表达的人来说，这是事实。不必担心脸面，确实我患有这样的病症）。什么病症了？

症状大致上如下：早上醒来先看看手机，然后上班路上骑自行车去地铁站时会打开B站或油管后台播放视频听。在地铁上做多邻国（5分钟一单元的西班牙语），然后就是看看短视频（西瓜、抖音、B站、Youtube Shorts），或者去虎扑看看球迷为球星之间撕逼（A: 你怎么又黑xxx球星? B: 怎么黑的？A: 贴xxxx球星说过的话或者比赛的视频。 B: 我擦，这也算黑？讲事实也是黑？A: 就算是这样，你也不能公开发出来）。等到到了公司，我打开电脑，开始工作预热（先看看人民日报、v2ex和油管），然后才进入工作状态。工作期间也会时不时刷下手机，上厕所拉屎也是，跟同事吃饭时候有时候也是。

下午3-4点时候锻炼五分钟，然后继续工作直到下班时，上了地铁，感觉累了，越是掏出手机打开短视频学习钓鱼的知识。然后出了地铁站，再次打开B站后台听听。直到到家了，跟老婆小孩吃饭。吃完饭，跟小孩玩一会，还是会忍不住看手机，总觉得有点累了，放松下。晚上倒垃圾时再锻炼五分钟。直到睡前，还会打开微信读书看会书，总是感觉有些累，但是又无法直接休息，这样重复第二天。

我发现当我感到越疲惫的时候，就越想刷短视频，特别是那种竖屏模式可以不断往下翻的视频。而且刷短视频很容易让我一下子就刷了很长时间，一旦停下来不刷的话，没有外力的影响，单靠自己的意志力感觉挺难的。最重要的是，我发现这样对于我的工作专注力和对孩子的耐心也没有那么好。特别是有一次带着孩子去游乐场，我发现大部分家长也都在刷短视频。

仔细想了想，上面提到的多巴胺成瘾要么是天然的那么是药物的，手机成瘾应该是一个比较特别的成瘾物了。就短视频而言，一方面它带有社交属性，通过通讯录容易加到朋友家人，这个本身带有愉悦感；另一方面它精心设计算法可以适配你的口味，而不断推同一个口味下面不同的视频。简单的说慢慢的缩小了给你带来愉悦感的事情，同时因为背后创造者的人是千差万别，导致虽然范围缩小到了一个区域，但是区域里的内容是不可预测的，随机的。比如我看的是钓鱼，当我搜索了几个之后，短视频算法立马知道我想看的是钓鱼，于是推了很多钓鱼的视频，但是你却不知道下一个是邓刚钓鲢鳙还是刘志强钓白条，台钓还是传统钓，正门钓法还是邪门钓法等等，是随机的，意料不到的。这种间歇性的随机不确定性正好可以刺激大脑不断释放多巴胺。

上瘾就像一个逐渐缩小能带给你快乐的事情范围的过程。

而工作或者跟小孩玩耍是可预测的按部就班的，这些不确定性相对比较低的活动带来的多巴胺的释放相对来说是少的。如果你像我一样，在工作之前和下班之后玩短视频得到了大量的多巴胺奖励，拔高了峰值，那么当你从事低多巴胺释放量的事情的时候，刚开始时你会感觉有点无聊空虚专注力下降，情绪等沟通的效率下降，直到过一会才恢复适应。

因此，在从事低多巴胺释放量的事情之前，可以尝试减少高多巴胺释放量的刺激，以减少对专注力、情绪等各方面的扰乱。

其次可以间歇性地试试吃饭时候吃饭，睡觉时睡觉。为什么吃饭时玩手机，睡前打游戏？可能是为了弥补工作太忙而缺少休息放松的时间。但是当吃饭时候玩手机，同时做两件事情太多太频繁，不仅影响吃饭的消化，忘记了品尝食物的本来味道（虽然很细微），而且其实耍手机并没有让我们更轻松，反而只是让我更疲惫，叠加的多巴胺刺激，更难以进入下午那些无趣工作琐事的状态。让大脑有一定的休息时间，而不是一直处于吸收状态。这个跟上厕所看手机也是一样的，我如果没有手机，我觉得无聊，于是我每次上厕所都带着，拿出来看是自然而然的。虽然没有像下面这哥们一样，玩游戏这么高多巴胺的“放松”活动：

来源：谁家这么狠？员工厕所“蹲坑”居然也有监控？！真相来了

有意思后来的是当我尝试专心跟我两岁半的小孩玩耍时候，我心想我确实感觉疲惫，但是刷手机只能让我得到虚假的放松和快乐，实质让我在现实中更加疲惫，为什么不试试放下手机书本电脑，一心一意跟他玩一会了？我开始观察他的每个动作，回应他的每个反.我们一起玩玩具，有时候他会让你帮忙一起搭一个东西，然后会有一阵子没有耐心了，又全部推倒，中间时不时有一些意想不到的妙语或鬼点子抑或是人类迷惑行为。有时候他会缠着我，有时候又会自己一个人玩得不亦乐乎。让我惊讶的是，我并没有感到特别疲惫（尽管身体可能仍有些许疲劳），内心却仿佛进入了某种状态。我不再思考其他事情或渴望其他东西，只是专注于当下与他一起玩耍，感受一种天然的人与人沟通的美妙体验。而且这个过程实际上并不会花费很长时间，你不要高估自己，他根本不会一直粘着你。也许只需要10分钟，他对你的热情就像刚刚搭的积木一样，过一会就可能要推倒重新来过了。

最后睡前微信读书其实问题不大，但是需要减少阅读这类《地下室手记》这种烧脑的书。

在某一周尝试调整一些行为来体验一下：

吃饭不看手机,就吃饭
拉屎就是拉屎（“道在蝼蚁，道在梯稗，道在瓦甓，道在屎溺”。）
一周里有一天喝白开水，不喝茶，凉茶，浓茶奶茶
删掉抖音等短视频
一周一天里是不带耳机听书
在地铁上不戴眼镜，观察人和物

习惯中的陷阱

当养成一个新习惯的时候，比如学习新语言或者学习双节棍，一开始容易幻想(visualize)自己成功后的样子，肯定对现在的语言水平和笨拙的手脚是嫌弃的。幻想成功后的样子会释放多巴胺，让人感觉快乐，即使此时还没达到目标。然而，过多这样的幻想可能会导致在进行了一两天后遭遇现实落差的打击，因为实际进展未达到幻想中的程度，容易引发情绪低落和放弃的想法。而此时，大脑渴望通过继续努力将失去的本钱赢回来，进而营造一个更大预期的渴望来释放多巴胺获得愉悦感。

不用沉浸在太多的思绪中，而是应该沉浸在体验中!

幻想太多太大容易造成不恰当的阶段目标设置，导致无法长期持续，因此尽量避免一开始就追求大量多巴胺释放的行为。在多邻国中的西班牙语学习环节中，可以设置每个单元的时长，例如5分钟、10分钟、30分钟和1小时。虽然从愉悦感来说，1小时肯定是最令人满足的，最贴近期待的奖励，但是现实中是非常难得，因为开始时刻因为预期的高多巴胺让你的理智能力有一定抑制，做出的评估极有可能是偏高的。在第一天，你可能会有很高的多巴胺释放，但到了第二天或第三天，你久会发现这个行为难，因为多巴胺释放没有一开始那么高，动力和情绪上都会下降，慢慢地会觉得这个坚持一小时是一种折磨。因此，我设置了最低的5分钟的学习时长。

同样的学习双节棍也是一样的原理，我先搜索了学习路线，然后从最基础的动作开始逐一学习。在学习的过程中，我专注于每个动作是否做到位，身体的协调性等方面。虽然我可能没有得到短期大量的多巴胺刺激，但我得到了最大化的沉浸体验。这种体验可能不是高多巴胺下的极致愉悦感，但小小的平静满足感从长期来看更容易坚持下去。

基准线天然的慢慢提高，可以自然的提高多巴胺水平，更容易长期提供动力

当一开始能感受到那种强烈预期而带来的兴奋，那种按耐不住跃跃越试的冲动时，心跳和肾上腺素加速时，可以试着不去行动，冷却一下（虽然很难，需要不断摸索实践），过一阵子等平静下来了（你可以观察身体和心率的变化），再去分析合理的预期和目标设置。（实际上，某些时候，做多巴胺奖励系统相反的事情，比如高多巴胺时，本能应该很爽，但是去做当时不那么爽的事情，相对适量巴胺释放的事情，比如发呆或者试着盯着远程某个物体观察它的每一个细节，同样的，在情绪低落或者某些低多巴胺时，也做这些，我个人感觉自己都会有一种奇妙无法用言语的感觉 - 类似传播学里冷新闻热处理热新闻冷处理）。

在多邻国学习中，当你每一天完成了一个单元时，你会得到一个动画效果，提醒你成功坚持了一天，动效是那么流畅丝滑，让你感觉要完成了一个阶段目标，因此感觉很好。同时在日历上，你可以看到连续打卡学习的记录，看到这么多天也会让你觉得视觉上很享受那根把日历上每个日子都连起来的线。这些都刺激着你让每天打卡，因为每个人或多或少都有点完美主义情节，带着点强迫症。

打卡动效非常灵动吸引人，当你坚持快1000天，刚这个日历连起来的视觉感觉就可以让你感觉到嗨

虽然我虽然没有中断过学习，但是最接近中断的一次是在上午以为我已经完成了这个单元，但实际上没有，直到晚上11点40分我才发现并赶紧完成了它。这是我最接近中断学习的一次体验。

后来我发现这种打卡形式能带来的满足感越多，那么反过来也可以说无形中它给设的套越紧，你不自觉的会担心中断了日历上那根完美的串联了所有日子的线要中断了，有一节缺失了，不完美了。这种情况下，越是坚持连续打卡的次数越多，那么实际心理的负担越重，你害怕的不是这个过程跳过了对于我长期目标有什么影响，而是这个形式上的瑕疵（我都坚持999天，就在第1000天的时候断了啊），这就像心理的一根刺，有些情况下会暗示自己的这个习惯的坚持已经失败，有的甚至上升到了对于自我的攻击。（摆烂了各位（地下室手记），习惯中断了，又回到了坏习惯的惯性中，完成心理还的加一句：你看我就是那么废，我都说了，你看，证明了吧，以后别瞎折腾）。这就有点像破窗效应, all or nothing，那么为了避免看到形式上的瑕疵，干脆放弃了整个后续习惯的坚持。（哪怕某些APP给你补救连续打卡的办法，其实心理上也会有上面类似的感受，你是没有办法欺骗你自己，无法fake it，这个也会潜意识的增加给心理一个负担）

就像上面我提到的接近中断体验，哪怕是五分钟，就是你拉屎或者坐地铁或者等饭菜快递好或者端上来的时间，就可以轻易做到。实际上中间有很多次，要么因为我头天赶飞机或者做卧铺失眠第二天状态不行，要么是有其他更重要的事情完全占据了这些日常打卡的时间点，要么是以为做了实际没做，要么放开后感染了两波新冠身体难受等等吧而差点点当天没有完成任务而没打卡成功。

看似形式上设计的精妙，打卡的体验和目标完成度的展示，越是吸引人，越是让人有成就感，那么反过来，你就越依赖它在形式上提供给你的快感，转变为另一种的多巴胺依赖了。而真正完成这个习惯坚持行为的过程中的实际的感受和体验反倒极有可能并没有带来形式上那样多的多巴胺刺激，因为任何习惯的坚持都是一个天然奖赏逐渐提高多巴胺基准水平的过程，这个多巴胺的释放量是一个甚至人都没有明显感觉的提高过程。这样一来，人对于习惯坚持的内在本质的追求，变成了对于习惯坚持的形式的追求，异化了。这就随着形式的坚持时间越久，那么每次打卡时多巴胺释放量会越来越大，达到了上瘾的地步，以至于某一天的跳过，人会觉得失落沮丧，不是因为没有能坚持习惯，而是因为感受到了就像突然没有烟可以吸了的那种多巴胺浓度下降带来的不适。

这就是有时候可以尝试下逆向多巴胺系统的思路，当感受到多巴胺系统给你带来不适，你要做高多巴胺释放的事情，用魔法击败魔法。这种情况是什么是高多巴胺的事情了？那就是它提醒了也许可以停下来重新审视自己的这个习惯，可能是它一直在自动化运行，变成了以原始的本能反射，你甚至都没怎么意识到了，而不是一个你认为你可以做的一个选择。你刚好可以停下来看看这阵子习惯坚持的效果怎么样，有什么需要修正调整的，方向没有走歪了，有没有走向形式化，自己的感受如何，总结一下，更好的服务于下一个阶段。这个刚好也是短期提高多巴胺的事情，突然打断它的可预测和确定性，增加了随机性，这样短期你就可以有足够动力客服破窗效应，而可以honestly express yourself，真实面对自己，不欺骗自己，进而可以更加自然进行一下习惯坚持的任务，你也可以给上述留出一定时间，让多巴胺回到正常基准，然后重新出发。

大脑是一套本能的原始的系统，大部分情况是都是自动化运行的，有时候很明显是会出bug，我们无法完全相信它，而类似的习惯日常化了，也会慢慢变成了自动化运行，所以我们需要时不时停下来检视它才能更好的发挥出它作用，科学是动态的，真理是不断发展的，所以需要保持敬畏的同时保持一定的怀疑。

去体验和感受

一种情绪和反应都有对应的同等的反面的情绪和反应。快乐的反面是痛苦，实际上大部分痛苦的来源（除了疾病）可能来自快乐的缺失，没有刺激物了，快乐感的量断崖式下跌，形成了所谓的痛苦。
快乐无法强行留住，无法持久，你不能抓住它让他别走，就像小孩子的抓摸不定的三分钟热度
习惯的开始最怕【三分钟热度】提前预支快感 - 设想visualize习惯养成之后的奖励。
习惯的中间最怕不是中断，而是中断造成形式上的不完美带来的疙瘩变扭，压过了最开始设置习惯的初心，沦为形式主义的傀儡。形式上奖励越能带来多巴胺的释放，愉悦感越高，那么它成为习惯坚持下去的最大的阻碍的可能性越大。

有一个有趣的现象是，当我们尝试养成习惯来获取某种技能状态或戒除某种有害行为时，通常是因为对当前状态不满意或感觉有提升的空间。因此，当习惯养成之后，我们期望收获一种更健康的行为、更高的技能或更好的状态，从而走向更好的自己。

然而，有一个有趣的观察是，似乎没有人通过习惯走向更坏的自己。这其实引起了一个很有意思的现象。以咖啡因为例，准确地说，它只能算是依赖，不能被归类为上瘾物质。但是，我们可以看看身边那些上瘾物质的人的行为，比如吸烟、酗酒、吸毒等等，假设将从开始到上瘾沉迷看作是一个习惯的培养过程。

一开始，他们肯定知道，如果有意识地将这个行为养成习惯并长期坚持，最终的状态将会是：抽烟会导致像烟盒背后那些恐怖的糜烂肺部照片所示的后果；喝酒会使人变成大脖子酒后发疯耽误事；吸毒更不用说了，直接走向死亡。他们会从这个习惯坚持后得到的“更好”状态获得预期的奖赏吗？会像买彩票一样释放大量的多巴胺吗？相比之下，他们情愿没有预期吗？

不可能吧！再者，他们会一开始就设定每天每个阶段很大的目标吗？毕竟这样可以更快速地达到目的，实现预期的多巴胺奖赏。这更不可能了！没有人第一天就抽10包烟、喝10瓶酒、吸10吨毒品，然后后面每天逐渐降低目标，相反，这些习惯一根根网上叠加。最后，在实施阶段任务时（比如抽一根烟、喝一杯酒、吸一点毒品），他们会幻想自己正在接近更好的状态，接近最终状态吗？不会的，他们甚至没有意识到这个习惯会有终点，就像刷牙洗脸一样再自然不过了。

此外，他们会完全沉浸在任务实施的过程中，真正专注于每一步中的技能和状态提升（一次次增加难度，从零开始），而不是在任务完成时打卡那一刻的满足感。他们更不会在某一天跳过了这个任务后感到灰心丧气，也不会因为形式上的连胜记录被终结而有一点点自我否定，相反，这会给他们带来更大的动力。

如果不知道从正面来如何构建一个习惯来达到更好的自己（表诠），那么从反面设想如何构建一个习惯来摧毁自己（遮诠）兴许能得到启发。

就像前面提到的，在高多巴胺浓度下面，大脑容易压制理智和情感中心不让他们活跃起来，而让我们陷入隧道视觉（tunnel vision）全心全意没头没脑般去追求让我们爽的事情。这个时候原始本能在全力冲刺，大脑中一切其他与认知相关的模块，都会降低活动频率，让开道路，动物性在大杀四方。在低浓度时，要严重感觉到某种情感上的痛苦混合无聊、空虚、消极，又让理智陷落于悲观主义，追求最快的最方便的后去原始的快感的方法而容易陷入到老的套路里。除了被动的被动物性牵着走，有什么办法了？

这两情况下，却是都可以引起身体的生理反应，心跳加快，呼吸急促或者某些身体的不适的痛苦等等，而且可以引起情绪情感的变化，这些可能是能给我们提供一个指导和方向。而这些也是是能让我们真实感受到自己活着的印记，通过这些其实每个人都有拥有的独一二位的印记，也许可以发现更深层次的自我，本能的自我，隐藏在自动运行机制下的自我，这个平常不容易察觉的自我，打破停滞，体会人生，体悟人性。

当1970年李小龙背部受伤躺在家一年，有可能从事不了武术之后， Later when writing to a friend about his back injury: “But with every adversity comes a blessing because a shock acts as a reminder to oneself that we must not get stale in routine.” It’s not the situation that’s the problem. It’s how you react to it.“

Bruce Lee #11 Walk On

可能是因为我足够变态，我能感受和观察那些快感或者痛苦或者无聊（除了生理疾病之外的），我不抗拒，当然不想着抓住它们留下来，但是也不会去驱赶它们，我就体验感受它们，专注的。当我放弃了思考和分析，卸下一切对抗阻力，带着好奇和真诚去感受着当下身体上的变化和思绪情绪上的激荡，感觉一种从未有过的鲜活的自我存在。真诚是永远的必杀技，沉默是金。

当然这还有一个意料不到的效果，这些情绪和反应通常不会持续很久，而可以比较容易快速的恢复到平静的状态，从正弦函数的波峰波谷回到正常的基准线。

在多巴胺高峰时候不要尝试人为去叠加或者加强刺激物，同时在多巴胺低估时，让时间慢慢去修复到多巴胺基准水平，不要为了逃避空虚痛苦去人为的寻求刺激物。去本能感受，不是去本能反应，自然而然地无为，通过有意识的慢慢探索和更深的自己了解学习，不断提高自己的觉察力，对于自己的身体情绪处于什么状态有一个认识一种敏感，能够突然的跳出来在一个距离之外观察自己，而不是自己被多巴胺绑架而事后短片一般的在迷局中不断地去寻求刺激物，才能更好的做出决策判断。《直视我!崽种!》

提高天然奖赏，也就是多巴胺基准线，需要持续摸索，是一个逐渐抬升的动态过程，让我们更敏锐地感受到日常幸福感。短期的大量多巴胺刺激释放会达到顶峰，可以全力去体验感受，但是不要觉得你自己通过主动寻求更大刺激而可以抓住和留下它，let it go，专注在持久的那些小的幸福感提升。像李小龙在电影《龙争虎斗》说的：这就好比一根指向月亮的手指，我们千万不要误将手指当成月亮，更不能专注于手指而错失了月亮的壮丽光辉。

Don’t React… Feeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeel!

NestJS+Prisma Dockerfile build optimization

2022-06-12T04:55:32+00:00

Recently, There was a NodeJS project developed by using web framework NestJS 7.0 and ORM Prisma 3.1.1 for the backend. The reason to choose those two tech as stack at that time seems quite oblivious: both support Typescript natively and allow for isomorphic development with React in the frontend. The backend has been divided into three modules roughly based on the platform to which its API is served: backend, frontend and frontend-emp. Apart from that, there are some shared libs, e.g. prisma schema defition , migration scripts and uitls. The structure of the codecase looks like following:

├── Dockerfile.backend
├── Dockerfile.frontend
├── Dockerfile.frontend-emp
├── README.md
├── apps
│   ├── dashboard
│   │   ├── src/**
│   │   └── tsconfig.app.json
│   ├── frontend
│   │   ├── src/**
│   │   └── tsconfig.app.json
│   └── frontend-emp
│       ├── src/**
│       └── tsconfig.app.json
├── assets
├── font
│   └── 微软�\233\205�\221.ttf
├── libs
│   ├── db
│   │   ├── prisma
│   │   |   ├── migrations/**
│   │   |   ├── schema.prisma
│   │   ├── src
│   │   └── tsconfig.lib.json
│   ├── shared
│       ├── src
│       └── tsconfig.lib.json
├── nest-cli.json
├── package.json
├── patches
│   └── exceljs+4.3.0.patch
├── tsconfig.build.json
├── tsconfig.json
├── webpack-hmr.config.js
├── yarn.lock
└── �\225��\215�说�\230\216.md

The database schema definitions are in the libs/db/prisma/schema.prisma, and the migrations are located under libs/db/prisma/migrations. A quick look a the DSL schema syntax:

generator client {
    provider        = "prisma-client-js"
    previewFeatures = ["filterJson"]
}

datasource db {
    provider = "postgresql"
    url      = env("DATABASE_URL")
}

model User {
    id        Int       @id @default(autoincrement()) //
    uid       String?   @default(uuid()) @db.Uuid // uid to reset password
    name      String // name
    ...
}

The connection string is obtained from the environment variable DATABASE_URL that was passed in. Every time your modify the schema.prisma, you need to run prisma migrate dev to let prisma client to check existing database schema and generate migration SQLs. After that, run prisma generate to generate a Typescript-based prisma client for your source code to use.

import { PrismaClient } from '@prisma/client'
const prisma = new PrismaClient()
// use `prisma` in your application to read and write data in your DB*
await this.prisma.user.findMany({...})

Prisma Concept Generating the client

Add Prisma to Dockerfile

Now we need to write the Dockerfile to generate Docker images, push them to Dockerhub, and deploy them into the Kubernets cluster. The process is done on the CI/CD platform(Jenkins) to make it fully automated. The cornerstone is the Dockerfile, and here is the initial code snippet:

FROM node:16-alpine

WORKDIR /home/node
COPY . /home/node

RUN yarn install
RUN yarn prisma generate

ENV NODE_ENV production
RUN yarn run build-frontend
EXPOSE 7021

CMD ["node", "dist/apps/frontend/main.js"]

No fancy stuff here. The first instruction is choose right base image to start with: NodeJS v16 based on the lightweight Linux Apline distribution. Dockerhub docker-node shows all the available versions and distributions, and it’s always helpful to click on those links to jump to their Dockerfine definitions on Github to get an idea of how their instructions are written. The following instructions are simple: copy the source code, run “yarn install” to install dependencies, run “prisma generate” to generate “@prisma/client”, run “nest build” to build the artifacts (which will be in the “dist” folder), and last but not least, run the command to start the whole NodeJS app: node dist/apps/frontend/main.js.

Prisma Generate

When you run the command docker build -t frontend-api -f ./Dockerfile.frontend . to build the image, you may notice an error being thrown at the line - RUN yarn prisma generate :

#9 3.978 Prisma schema loaded from libs/db/prisma/schema.prisma
#9 4.849 Error: Unable to require(*/home/node/node_modules/prisma/libquery_engine-linux-musl.so.node*)
#9 4.849  Error loading shared library libssl.so.1.1: No such file or directory (needed by /home/node/node_modules/prisma/libquery_engine-linux-musl.so.node)

From the stack trace, it looks like there are two points worth noting:

*  The shared library `libssl.so.1.1` could not be loaded.
*  This liblibrary is referenced by `node_modules/prisma/libquery_engine-linux-musl.so.node`

The “libssl.so.1.1” library is probably related to SSL from its name, particularly on Linux, and it’s dynamically linked at runtime by “libquery_engine-linux-musl.so.node”. After a quick search, we found some Github issues on the OpenSSL repository Openssl can’t find libssl.so.1.1 and libcrypto.so.1.1 · Issue #19497 · openssl/openssl (github.com) and the Prisma repository Support OpenSSL 3.0 for Alpine Linux · Issue #16553 · prisma/prisma (github.com) that suggest the operating system we’re running on may not include the OpenSSL dependency or we may have OpenSSL, but its version doesn’t satisfy the requirements.

To double-check, we can examine the base image of our Dockerfile, which is “node:16-alpine” - docker-node/16/alpine3.15/Dockerfile:

FROM alpine:3.15
ENV NODE_VERSION 16.13.1
...
  && apk add --no-cache --virtual .build-deps-full \
        binutils-gold \
        g++ \
        gcc \
        gnupg \
        libgcc \
        linux-headers \
        make \
        python3 \
  && yarn --version
...

Previously, we knew that we wanted version 16 of NodeJS running on Linux Apline, but we didn’t know the exact versions of NodeJS and Alpine. By examining the Dockerfile, we can see that it uses NodeJS version 16.13.1 and Alpine version 3.15. The next part of the Dockerfile installs basic development dependencies: Node and Yarn. However, there are no instructions to install the OpenSSL library, which is the root cause of the problem.

The solution to this issue is straightforward: install OpenSSL version 1.1 before the prisma generate instruction by using APK installer.

FROM node:16-alpine
WORKDIR /home/node
COPY . /home/node

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories #change source repo to AliCloud mirror, only needed in China
RUN apk update  # updating of the indexes from remote package repositories
RUN apk add openssl1.1-compat # install openssl 1.1

Rrerun the docker build, now it works!

Prisma Generate - How it works

Even though the problem seems to be solved, there remains a mistery to me: does Prisma need OpenSSL to support SSL connections with the database, and what is the role of node_modules/prisma/libquery_engine-linux-musl.so.node in prisma generate?

It turns out that the Prisma documentation has a section dedicated to this: Generating the client (Concepts) (prisma.io). When you run `prisma generate, prisma client package (“yarn add @prisma/client”) will generate a PrismaClient with three components:

Typescript definitions(index.d.ts)
Javascripts code(index.js) - implementaion of the Prisma Client API.
Query engine(libquery_engine-xxx.xx.node) under path node_modules/.prisma/client.

Recall that you first need install the “yarn add @prisma/client” package. That @prisma/client consists of two key parts:

* The @prisma/client module itself, which only changes when you re-install the package:  *node_modules/@prisma/client*
* The .prisma/client folder, temporary, which is the default location for the unique Prisma Client generated from your schema: *node_modules/.prisma/client*

import { PrismaClient } from '@prisma/client'
const prisma = new PrismaClient()
// use `prisma` in your application to read and write data in your DB*
await this.prisma.user.findMany({...})

When we include the PrismaClient in the source code and call this.prisma.user.findMany, the prisma client sends this command to the query engine. Then the query engine translates it to SQL queries and sends them forward to the database. Once the database has the query result, it sends it back to the query engine, which translates those raw data to plain JavaScript objects and sends them to the prisam client.

Prisma Concept Generating the client

Prisma Client uses a query engine to run queries against the database. This query engine is downloaded when prisma generate is invoked and stored in the output path together with the generated Client.

The query engine is tailed to be compiled and built on that platform to achieve better performance.

It is named query-engine-PLATFORM or libquery_engine-PLATFORM where PLATFORM corresponds to the name of a compile target. Query engine file extensions depend on the platform as well. As an example, if the query engine must run on a Darwin operating system such as macOS Intel, it is called libquery_engine-darwin.dylib.node or query-engine-darwin

It does match the name on my macOS Intel. But how about the name on Linux Alpine 3.15 docker image? We could find it out in Prisma schema API (Reference):

PLATFROM name is linux-musl, and it requires OpenSSL 1.1.x to be installed. The full name would be libquery_engine-linux-musl.so.node - exactly the name that shows up in the error trace. However, after Primsa 4.8.0, prisma has given better support for OpenSSL 3.0: Support OpenSSL 3.0 for Alpine Linu.

install OpenSSL 1.1 -> download query engine for platform specific -> prisma generate

How to run Prisma Migrate Deploy

How do you run migration sqls on a production environment? First you need have the libs/db/prisma/migration folder. Deploying database changes with Prisma Migrate recommends that you do not run it locally againist the production database. Instead, run prisma migrate deploy during the release phase. It requires access @prisma/client dependence, so just move it from devDependencies to the production dependencies section in your package.json.

- CMD ["node", "dist/apps/frontend/main.js"]
+ ENTRYPOINT [ "npm" ,"run"]
+ CMD ["start:prod_frontend"]

in your package.json, add following:

  "scripts": {
    "build-frontend": "nest build frontend",
    "start:prod_frontend": "yarn prisma migrate deploy && node dist/apps/frontend/main",
  }
  "prisma": {
     "schema": "libs/db/prisma/schema.prisma"
   }

Dockerfile optimization

In the previous step, we integrated Prisma into the Dockerfile and are now able to run it. However, the Docker image that final build produces is quite large, 1.53G, and the build time is slow. This is not ideal for a CI/CD pipepline.

server ➤ docker images                                                                                                                                                                                                                                              REPOSITORY     TAG       IMAGE ID       CREATED             SIZE
frontend-api   latest    93727cf78c85   About an hour ago   1.53GB

Before diving into optimization, let’s take a look at how Docker builds images based on the Dockerfile.

FROM node:16-alpine  #Layer n
USER root            #Layer n+1
WORKDIR /home/node   #Layer n+2
COPY . /home/node    #Layer n+3
RUN yarn install     #Layer n+4
RUN apk update       #Layer n+5
RUN apk add openssl1.1-compat #Layer n+6
RUN yarn prisma:generate      #Layer n+7
ENV NODE_ENV production       #Layer n+8
RUN yarn run build-frontend   #Layer n+9
EXPOSE 7021                   #Layer n+10

ENTRYPOINT [ "npm" ,"run"]    #Layer n+11
CMD ["start:prod_frontend"]   #Layer n+12

The process is like a game of Jenga, where each layer is placed on top of another layer like a stack. From node:16-alpine is layer n, COPY . /home/node is layer n+1， which comes after the layer n. Each instruction corresponds to a layer, and finally, the docker images is just a bunch of layers stacked togher. The most commonly used instructions are COPY and RUN. COPY deals with file-related operations, such as copying files from current directory of the build context to the layer in docker image, while RUN is used for executing shell commands or scripts.

The whole process takes a considerable amount of time when built for the first time. However, the second time, it will go much faster as Docker provides a cache for repetive work to speed up the build process. How does the cache work? It simply remembers whether you made changes to the current layer. If any changes are detected in the layers, itself and following layers will get a rebuild. Like the Jenga game, all blocks will get rebuild on top of where that block got pulled.

Clearly, in the above Dockerfile, if you modified ./libs/db/prisma/schema.prisma file to, e.g, add a new table or table field, it will trigger a rebuild starting from Layer n+3 - COPY . /home/node since ./libs/db/prisma/schema.prisma is under the current build context directory. Docker is smart enough to spot that change and trigger a rerun of that instruction and all instructions below - invalidating the build cache.

You may be wondering that if you only changed schema.prisma, it should just trigger prisma to regenerate its client and rebuild the nest project and nothing more. However, it also triggers “yarn install”, which you didn’t even touch. Yeah, that Dockerfile is not very efficient. There are a couple of ways to improve it:

1.Order your layers

Any change in the source code, not just package.json or yarn.lock, will trigger Docker to reinstall all packages for the project. Acutally, the COPY . /home/node happens to do two things at one time: 1. copy project dependencies, 2. copy source code. Those two not only comes with differnet responsibilities but also comes with different modify frequencies. The time you modify your project dependencies is significantly less than the time you modify the source code. So the Dockerfile could be changed to:

COPY package.json yarn.lock .yarnrc .
RUN yarn install #install project  packages
COPY . . # copy source codes
RUN apk update
RUN apk add openssl1.1-compat
RUN yarn prisma:generate
ENV NODE_ENV production
RUN yarn run build-frontend

Different modify frequencies sginal possible different modules of reponsibilities. Most project Dockerfiles could be broken down into the following stages:

Prepare the operating system and base image - this step is required only once.
Install system-level libraries - this step is also usually required only once.
Install project-level dependecies - this step may need to be repeated if packages are added, removed, or updated
Modify your source code in codebase - this step is typically done on a daily basis.
Build your project
Run it

With this order in place, we could change the Dockerfile to:

# 1.base image OS
FROM node:16-alpine
USER root
# 2.install system level libs
RUN apk update
RUN apk add openssl1.1-compat
# 3. install packages
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc /home/node
RUN yarn install

# 4.copy source code, generate prisma client
COPY . .
RUN yarn prisma:generate

# 5. build
ENV NODE_ENV production
RUN yarn run build-frontend

# 6. run
EXPOSE 7021
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

2.Keep layers small

Generally, the instruction COPY . . indicates there is some bad smell in the Dockerfile. Even if you use a .dockerinogre file to tell Docker to skip certain files, like common files or directories such as node_modules and dist, it is recommended that you explicitly write down what files you want to include in the instruction. Explicitly naming what you want to include helps you think about what you’re gonna use it for. Typically, coping files is just the first step, and it is usally used as an input for following steps.

# 4.copy source code, generate prisma client
COPY . .
RUN yarn prisma:generate

The step that takes input is RUN yarn prisma:generate. What does it do? it takes a package.json file and looks up where the schema.prisma is located. Once it finds the path to schema.prisma, aka libs/db/prisma/schema.prisma, it loads that file and then uses @prisma/client package to generate prisma client files. So this instruction takes two files: package.json and libs/db/prisma/schema.prisma. Therefore, the input of this instruction just needs to copy those two files, not the all files in codebase.

* command: yarn prisma generate
* input:   package.json and libs/db/prisma/schema.prisma
* output:  node_modues/.prisma/client

The same idea applies to the RUN yarn run build-frontend instruction, which runs nest build frontend to build frontend-related source code to the dist folder. So source code doesn’t need include apps/backend and apps/frontend-emp, it just needs frontend and shared libs, alone with package.json and nest-cli.json.

* command: yarn run build-frontend
* input:   apps/frontend, libs, package.json and nest-cli.json
* output:  dist

# 4. prisma generate
COPY libs/db/prisma/schema.prisma ./libs/db/prisma/schema.prisma
RUN yarn prisma:generate

# 5. build
COPY apps/frontend ./apps/frontend
COPY libs ./libs
COPY nest-cli.json tsconfig.json libs .
ENV NODE_ENV production
RUN yarn run build-frontend

3. RUN: Combine commands together wherever possible

When Docker build tries to decide whether a layer needs to be rebuilt or not, it checks with the cache. If the cache was hit, then there is no need to rebuild this layer. If not, it will run the instructions and rebuild the current layer and following layers. How does the cache work? How does this calculation process work ? Best practices for writing Dockerfiles: Leverage build cache mentions that Docker will do a file read and calculate its content checksum and compare it with the one the exsting image for the COPY insturction, and do a command string literal equal check for the RUN instruction. Let’s take a look at the common command apk/apt-get for RUN instructions:

RUN apk update
RUN apk add openssl1.1-compat

Initially the instructions seems okay. apk update will fetch indexs(e.g name, versions, etag) of packages from remote repositories. apk add openssl1.1-compat will then get installed with the latest version from the indexes.

Suppose at this time, the latest versions of libraries in apk repos are:

openssl1.1-compat=1.0 wget=1.0 curl=1.0

The current latest versions of openssl1.1-compat is 1.0, and it will get installed. Then the whole thing gets forgotten for ten years, and it needs to add another library: wget.

RUN apk update
RUN apk add openssl1.1-compat wget

Ten years after, the apk repos is like:

openssl1.1-compat=1.9 wget=1.9 curl=1.9

To your surpise, the versions in the final docker image that got built are openssl1.1-compat (1.0 old) and wget (1.0 old too). What happened here is that Docker saw ("RUN apk update") in dockerfile == ("RUN apk update") in docker image - two command strings equals! Then it picked up the cache and skipped actually rebuilding this layer. As a result, it didn’t run “apk update” to actuall y update local indexes from remote repo. It ended up with old and stale packages info in the local. The next insturction ("RUN apk add openssl1.1-compat wget") in dockerfiler != ("RUN apk add openssl1.1-compat") in docker image, then the cache was invalidated, and this layer needed to be rebuilt, hence running apk install with wget version 1.0.

So if it is just a command string literal compare, we could just merge those two into one command:

- RUN apk update && apk add openssl1.1-compat
+ RUN apk update && apk add openssl1.1-compat wget

Now, Docker sees apk update && apk add openssl1.1-compat != apk update && apk add openssl1.1-compat wget, it will invalidate the cache and rebuild the layer, which is called cache busting. Now apk update will fetch the latest indexes, and apk install will install the latest version. The version of wget will be 1.9. However, the version of openssl1.1-compat is 1.9 too. This bump from 1.0 to 1.9 might accidentally break our code and cause some troubles.

The rule of thumb is to lock down the version info for packages. This is called version pinning, you could search for what versions those packages have for that architecture. For example, for apk, you could go Alpine Linux packages or apk search to look up version informations. For openssl1.1-compat, its version we picked is 1.1.1t-r0.

RUN apk update && apk add openssl1.1-compat=1.1.1t-r0 wget=1.9

That’s better. let’s examine this command:

* command: apk update && apk add openssl1.1-compat=1.1.1t-r0 wget=1.9
* input:   none
* output:  openssl1.1-compat lib and wget lib, plus cache under /var/cache/apk/

It has some byproducts from the typical package management tools - the cache. We also need clean that up as it is not necessary.

RUN  apk update && apk add openssl1.1-compat=1.1.1t-r0 && rm -rf /var/cache/apk/*

or add –no-cache flag to apk install:

RUN  apk update && apk add --no-cache openssl1.1-compat=1.1.1t-r0

Given the installion of system libraries are pretty much one-off, you rarelly need to bother to use cache between build to reuse it. However, if you really want to cache between builds to speed up installation proces, you could try leveraging Docker builder’s new build tech BuildKit:

RUN \
    --mount=type=cache,target=/var/cache/apk \
    apk update && apk add openssl1.1-compat=1.1.1t-r0

It is like mounting a dedicated volume for that, so the cache will be preserverd bewteen builds. This will be elaborated in following yarn/npm section.

4. NPM/Yarn install packages

when Docker build a layer on RUN yarn install, it is quite slow and takes quite a big size 1.19G out of 1.53G total final docker image size. Just like adding system-level packages with apk add, yarn install also leverages cache.

* command: yarn install
* input:   package.json yarn.lock .yarnrc
* output:  node_modules, plus cache

Put it simply, when yarn install happens, it fetches the package’s compressed file and a metadata(name,version,modified time etc) file from npm registry, then saves those to the local directory that environment variable YARN_CACHE_FOLDER points to, finally decompressing it to the node_modules folder.

RUN yarn cache dir && yarn install && yarn cache clean

The output is clean now, but there is problem with this approache. If you need to add/update/delete a package in the package.json and updates its yarn.lock file, every time you build the docker image, it will fetch from remote repository, save to the cache folder, and move to the node_modules folder for all packages, even if you just want to add one new package. With the help of proper ordering of the layers and relatively low frequency of changing project dependencies, this problem has alredy been mitigated to a great extent.

However, there are possible chances that you need update packages, and to make it even worse, like moments for some critical bug-fixing for live running environment, this would put a lot of congnitive load on you. Therefore, it is helpful to dig more into it and figure out why and how it could be improved.

BuildKit

One way to address this is to leverage the BuildKit, the new Docker builder, to act like a mounted volume for caches between builds. BuildKit not only tracks content mounted for specific operations but also can parallelize building independent build stages in Multi-stage builds. Here, we’re going to use RUN –mount type=cache to mount the yarn install operation:

RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install

To do this, set the mount type to cache and target to some directory, then overwrite yarn cache folder with that directory by setting the YARN_CACHE_FOLDER enviroment variable for yarn install. My Docker Desktop on Mac is 4.16.2 (95914), which has built-in support for Buildkit and is on by default. You should check which docker version you have installed and may need to explicitly configure it to enable Buildkit.

<missing>      2 minutes ago    RUN /bin/sh -c YARN_CACHE_FOLDER=/cache/yarn…   628MB     buildkit.dockerfile.v0

This way, the cache - yarn install byproduct - will not be included in the output thus will not go to a layer in docker image. We dont’ need to clean the cache after yarn install. The result is a huge drop in terms of docker image size: from 1.19G to 628M. Not bad at all!

yarn install –perfer-offline

Le’t try add brand new package, yarn add underscore@1.13.6, then build:

=> [stage-0 5/9] RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install                                                                                                                                                                   114.3s

roughly 114 seconds, like 2 minutes. Only add this new package understore, the time is kinda too long for this simple task.

It is time to take a closer look at how yarn install works. By default, everytime it runs yarn install, it will use the version,name and integrity to check if there is a matche in the local cache.

if there is a match, it will send a 304 request to check if the one in local cache is stale or not; if stale, it will download new package info. Otherwise, it will just use the one in local cache
if there is no match, it will fetch latest data from remote repository to the local cache

flowchart TD a0[["yarn install"]] a1{"has internet connection?\n是否联网？"} a2{"matched in local cache?\n本地缓存是否命中?"} a3{"remote check if stale?\n远程访问校验是否过期?"} a4{"fetch \nand update cache \nthen unzip to node_modules\n拉取更新本地缓存\n并解压到node_modules"} a10["use cache\n使用本地缓存"] a20["fetch and update cache\n远程拉取并跟新本地缓存"] a30["use cache\n使用本地缓存"] a0 --> a1-->|YES|a2-->|YES|a3-->|YES|a4 a1-->|NO|a10 a2-->|NO|a20 a3-->|NO|a30

Those extra 304 requests may be the culpritsm as the more packages you have, the longer it takes. Is there any way to adjust the fetch behavior?

According to the npm blog NPM v5.0.0 and PNPM, there’s an option for yarn install - --prefer-offline - that makes npm skip any conditional requests (304 checks) for stale cache data and only hit the network if something is missing from the cache.

=> [stage-0 5/9] RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install --prefer-offline                                                                                                                                                   61.7s

By using this option, we managed to reduce the installation time from 114.3s to 61.7s :)

If you want to go completely offline, you could use the methods outlined in the blog Running Yarn offline to configure yarn-offline-mirror and yarn-offline-mirror-pruning to pack the dependecies to zip/tarball and upload to souce control. When you build docker image, yarn could directly load packages from the offline mirror folder and move them to the node_modules folder.

5. Use multi-stage builds

Let’s recall the steps we breakdown in previous step:

Prepare the operating system and base image - this step is required only once.

Install system-level libraries - this step is also usually required only once.

Install project-level dependecies - this step may need to be repeated if packages are added, removed, or updated

Modify your source code in codebase - this step is typically done on a daily basis.

Build your project

Run it

When putting all these steps together, we need to keep in mind the input and output. THe output could have some unexpected byproduct that probably involves some custom cleanup shell scripts. Just think about the final image and the running part - if it’s in Golang, it only needs a binary executeable to run. It is that simple. However, in NestJS project, it requires more files - three parts: dist folder, node_modules folder, and package.json file. Ideally, we only need those three layers for each one. We should encapsulate the implementation details and have a clear interface or high-level abstraction just like Object-oriented programming (OOP). Any concret implementation details like intermediate or byproducts are not my concerns and shouldn’t be!

Multi-stage builds allow you to drastically reduce the size of your final image, without struggling to reduce the number of intermediate layers and files.

Multi-stage builds is what we want here. Currently, we have a naive sequence of stages:

devDependecies(yarn install) ->  .prisma/client(prisma gnerate) ->  dist (nest-build) -> prodDependecies(yarn install --production)

flowchart TD A[base] -->|yarn install| B{Dev Dep} B --> |prisma generate|H(node_modues/.prisma/client) H --> |nest build|I(dist) I --> |yarn install --production| J{Prod Dep} J -->|node_modules prod| D[Final output]

FROM node:16-alpine as dev-dep
USER root
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc .
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --prefer-offline

FROM node:16-alpine as prisma-binary
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache openssl1.1-compat=1.1.1t-r0
WORKDIR /home/node
COPY --from=dev-dep /home/node/node_modules/ ./node_modules/
COPY package.json yarn.lock .yarnrc .
COPY libs/db/prisma/schema.prisma ./libs/db/prisma/schema.prisma
RUN yarn prisma:generate

FROM node:16-alpine as nest-build
WORKDIR /home/node
COPY --from=prisma-binary /home/node/node_modules/ ./node_modules/
COPY apps/frontend ./apps/frontend
COPY libs ./libs
COPY nest-cli.json tsconfig.json libs .
RUN yarn run build-frontend

FROM node:16-alpine as prod-dep
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc ./
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --production --ignore-scripts --prefer-offline

FROM node:16-alpine
ENV NODE_ENV production
WORKDIR /home/node
COPY  --from=prod-dep home/node/node_modules/ ./node_modules
COPY  --from=dev-dep /home/node/dist ./dist
COPY  --from=dev-dep /home/node/node_modules/.prisma ./node_modules/.prisma

COPY libs/db/prisma/ ./libs/db/prisma/  # need prisma migrations
COPY package.json yarn.lock .yarnrc ./  # yarn prisma:migrate

EXPOSE 7021
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

The final stage is one where its layer will get to the final docker image. We can use COPY –from=prisma-binary to explictly copy the output from previous stage to the current stage, and the fromStage is ephermeral - all those intermediate stages will not brought to the referring stage. That way, it would create minimal layers in the final docker image , hence reducing the docker image size.

You may wonder why we need to have a separate stage for prod-dep to yarn install production dependencies again, why not just copy all dependeices from dev-dep, then run npm prune to prune out any development dependecies. The reason is that there isn’t any good way to do that in yarn, as noted in this issue npm prune equivalent behavior · Issue #696 · yarnpkg/yarn (github.com) :(

The docker image’s size has dropped to 420MB. The node_modules folder is 218MB and .prisma/client is 81.7MB

7 minutes ago   COPY /home/node/node_modules/.prisma ./node_…   81.7MB    buildkit.dockerfile.v0
7 minutes ago   COPY home/node/node_modules/ ./node_modules …   218MB     buildkit.dockerfile.v0

History of size changes of docker image:

6. Dockerfile user and permission

By default, Docker containers run as root. That root user is the same root user of the host machine, with UID 0. This could be leveraged by hackers to do malicious attack on host machine. Given the default user is root, we could delete this instructions: USER root. Best practices for writing Dockerfiles suggest using a non-root user for services that don’t need privilege. When creating user and group, consider an explicit UID/GID. We can take some inspiration from the base image (node-16:apline) we used here: docker-node/Dockerfile at cd7015f45666d2cd6e81f507ee362ca7ada1bfee · nodejs/docker-node (github.com)

FROM alpine:3.17

ENV NODE_VERSION 18.14.2

RUN addgroup -g 1000 node
&& adduser -u 1000 -G node -s /bin/sh -D node

The permission of /home/node directory is under the node user. In order to set the running user for the container, you could put that USER instruction before the entrypoint and cmd:

EXPOSE 7021
USER node  # switch user from default root to node (created in base image)
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

Run docker build again, it throws the following error:

@tuo.local ➜ server rvm:(system) git:(uatFt) ✗ docker run 09ba84abb858
$ /home/node/node_modules/.bin/prisma migrate deploy
Error: Can't write to /home/node/node_modules/@prisma/engines please make sure you install "prisma" with the right permissions.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It looks like we don’t have sufficient permission to write to the node_modules directory. yarn install is running under the root user, but the user who runs prisma migrate deploy when container is up is node. User node doesn’t have enough permission to write to a directory owned by user root.

To verify our thought, we’d better to connect to shell of the container(docker exec -it xxx /bin/sh) and run ls -al to check node_modules folder’s permission. But the container is down when it is just starting, so we didn’t even have a chance to connect to its shell. What we could do here is, in development debug process, we change the CMD/Entrypoint to some script could hang the container forever to prevent it from quitting too early - something like tail -f. Also we need install the bash system library through apk so we could connect to the bash program and run commands.

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache bash  #install bash
EXPOSE 7021
USER node
#ENTRYPOINT [ "npm" ,"run"]
#CMD ["start:prod_frontend"]
#dev purpose
ENTRYPOINT ["tail", "-f", "/dev/null"]  #this will keep container quit too early

Now run docker exec -it 916c42e68c3b /bin/bash and list the folders:

916c42e68c3b:~$ ls -al
-rw-r--r--    1 root     root           430 Feb 24 10:40 .yarnrc
drwxr-sr-x    3 root     node          4096 Feb 25 15:19 dist
drwxr-sr-x    3 root     node          4096 Feb 26 03:53 libs
drwxr-sr-x    1 root     node          4096 Feb 25 15:19 node_modules
-rwxr-xr-x    1 root     root          8230 Feb 26 03:13 package.json
-rw-r--r--    1 root     root        376897 Feb 25 14:05 yarn.lock

916c42e68c3b:~$ ls -al node_modules/@prisma/engines
-rw-r--r--    1 root     node           537 Feb 25 12:40 README.md
drwxr-sr-x    2 root     node          4096 Feb 25 15:18 dist
drwxr-sr-x    2 root     node          4096 Feb 25 15:18 download
-rw-r--r--    1 root     node          1384 Feb 25 12:40 package.json

As we could see, the group node only has permission to read for node_modules/@prisma/engines, but no write permission. Here is a trap: instead of just recklessly adding chmod -R g=rwx ./node_modules/@prisma/engines to Dockerfile to “fix” it, you need think about what it needs to write to this node_modules/@prisma/engines folder? Isn’t from the official documents (as we list in above section) saying that it only changes when you re-install the package ? Is node_modules/.prisma/client supposed to be changed even though we did run a different command prisma migrate deploy?

The @prisma/client module itself, which only changes when you re-install the package: node_modules/@prisma/client

The .prisma/client folder, temporary, which is the default location for the unique Prisma Client generated from your schema: node_modules/.prisma/client

Hmmmm, here is the good part about our dev debug trick to get our container running to debug our startup script or command. We could just run exact same command in the bash to see what’s going on there. So we run yarn prsiam migrate deploy in the bash directly inside the container:

We encountered the same error, which is good, as it appears that the folder node_modules/@prisma/engines is missing the query engine file libquery_engine-linux-musl.so.node when compared to my local node_modules folder. I guess this is because the query engine binary file is quite large and needs to be fetched over the internet, which is a one-off event. However, the .prisma/client folder is frequently updated as users may modify the schema.prisma file many times during development phase. They may even delete the entire folder and regenerate the client, causing the slow fetching of the large binary file which negatively impacts the development experience and added unnecessary cognitive load on developer. Therefore they made a copy inside @prisma/client to just in case .prisma/client needs it.

A quick workaround would be copy @prisma from devDep stage and remember to install OpenSSL 1.1.

COPY  --from=dev-dep /home/node/node_modules/.prisma ./node_modules/.prisma
COPY  --from=dev-dep /home/node/node_modules/@prisma ./node_modules/@prisma
COPY libs/db/prisma ./libs/db/prisma
COPY package.json yarn.lock .yarnrc ./

RUN apk update && apk add  --no-cache openssl1.1-compat=1.1.1t-r0
EXPOSE 7021
USER node
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

Run again, no problem.

docker run d61f5e95f91754a3ad8d71cf232a6f09a5a81025aa767b6a8170158bc132165b
yarn -v && printenv && yarn prisma migrate deploy && node dist/apps/frontend/main

$ /home/node/node_modules/.bin/prisma migrate deploy
Prisma schema loaded from libs/db/prisma/schema.prisma
Datasource "db": PostgreSQL database "simeProdFeb21bk1", schema "public" at "host.docker.internal:5432"

263 migrations found in prisma/migrations

No pending migrations to apply.
Done in 1.88s.
[Nest] 17   - 02/26/2023, 4:49:59 AM   [NestFactory] Starting Nest application...

We pre-generate all the TS, JS and binary files for the prisma client to use. The advantage is that when container starts, yarn prisma migrate deploy doesn’t need to fetch over internet and generate clients, so the startup time is fast. The disadvantage is that the size of docker image would be bigger as we include two copies of binary target.

Another way to avoid copying those two folders into the Docker image is to have the script yarn prisma migrate deploy check the NodeJS-API and download the binary target file when starting up, and then generate clients.

- COPY  --from=dev-dep /home/node/node_modules/.prisma ./node_modules/.prisma
- COPY  --from=dev-dep /home/node/node_modules/@prisma ./node_modules/@prisma
+ RUN chmod -R g=rwx ./node_modules/@prisma/engines #permission
+ ENV PRISMA_BINARIES_MIRROR http://prisma-builds.s3-eu-west-1.amazonaws.com #speed up in China

The pros are the size of final docker image is smaller. The cons are it need download binary file and generate clients - both are time consuming.

Other improvement tips

Keep base images up-to-date for better performance and security.
Minimize the size of each layer as much as you can. For instance, considering using tools like depcheck - npm (npmjs.com) to scan the dependencies and remove any unused ones. Also, be careful when deciding whether a new dependecy should be added to the development phase or production phase.
Keep packages or dependencies up-to-date. npm-check-updates - npm (npmjs.com) can help detect any version updates in npm registry. For example, it can show that we have a version update for Prisma:

prisma ^3.1.1 → ^4.10.1

This update can resolve OpenSSL issues and also improve query efficiency while reducing the binary target size.

Release 4.10.0 · prisma/prisma (github.com)

Smaller engine size used in Prisma CLI

In 4.8.0, we decreased the size of the engines by ~50%, which significantly impacted Prisma Client, especially in serverless environments.

In this release, we’ve reduced the size of Prisma CLI by removing the Introspection and Formatter engines. The introspection functionality is now served by the Migration Engine. A cross-platform Wasm module has entirely replaced the Formatter Engine. This reduces the overall installation size for Prisma CLI.

The size of query engine has dropped 50%. That is impressive!

After applied with above optimization methods, the docker image’s size has reduced to 360M!

The sizes of node_modules and prisa layer has been shrinked a lot:

Final Dockerfile

FROM node:16-alpine as base
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache openssl1.1-compat=1.1.1t-r0

FROM base as dev-dep
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc ./
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --frozen-lockfile --prefer-offline

COPY libs/db/prisma/schema.prisma ./libs/db/prisma/schema.prisma
RUN yarn prisma:generate
COPY apps/frontend ./apps/frontend
COPY libs ./libs
COPY nest-cli.json tsconfig.json libs .
RUN yarn run build-frontend

FROM node:16-alpine as prod-dep
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc ./
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --frozen-lockfile  --production --ignore-scripts --prefer-offline

FROM base
ENV NODE_ENV production
WORKDIR /home/node
COPY  --from=prod-dep home/node/node_modules/ ./node_modules
COPY  --from=dev-dep /home/node/dist ./dist
COPY  --from=dev-dep /home/node/node_modules/.prisma ./node_modules/.prisma
COPY  --from=dev-dep /home/node/node_modules/@prisma ./node_modules/@prisma
# need migrations
COPY libs/db/prisma ./libs/db/prisma
COPY package.json yarn.lock .yarnrc ./

EXPOSE 7021
USER node
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

#dev purpose
# RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache bash
# ENTRYPOINT ["tail", "-f", "/dev/null"]

References

NestJS+Prisma Dockerfile 构建优化

2022-06-10T04:55:32+00:00

最近接触一个项目是用NestJS7.0和Prisma3.1.1作为技术栈来开发的后端，用这两个原因很明显：原生支持Typescript，前后端都可以用上JS的技术栈，后端相对来说更符合上云这个轻量化要求。NestJS这边的大概有三个模块: backend, frontend和frontend-emp, 大体还是根据面向的前端不同做的粗糙的划分， libs里面有一些公共的组件库，比如prisma关联一些迁移脚本和数据库表的定义。

├── Dockerfile.backend
├── Dockerfile.frontend
├── Dockerfile.frontend-emp
├── README.md
├── apps
│   ├── dashboard
│   │   ├── src/**
│   │   └── tsconfig.app.json
│   ├── frontend
│   │   ├── src/**
│   │   └── tsconfig.app.json
│   └── frontend-emp
│       ├── src/**
│       └── tsconfig.app.json
├── assets
├── font
│   └── 微软�\233\205�\221.ttf
├── libs
│   ├── db
│   │   ├── prisma
│   │   |   ├── migrations/**
│   │   |   ├── schema.prisma
│   │   ├── src
│   │   └── tsconfig.lib.json
│   ├── shared
│       ├── src
│       └── tsconfig.lib.json
├── nest-cli.json
├── package.json
├── patches
│   └── exceljs+4.3.0.patch
├── tsconfig.build.json
├── tsconfig.json
├── webpack-hmr.config.js
├── yarn.lock
└── �\225��\215�说�\230\216.md

Prisma中关于表的定义是在libs/db/prisma/schema.prisma里，DSL大概写出来是这样的：

generator client {
    provider        = "prisma-client-js"
    previewFeatures = ["filterJson"]
}

datasource db {
    provider = "postgresql"
    url      = env("DATABASE_URL")
}

model User {
    id        Int       @id @default(autoincrement())
    uid       String?   @default(uuid()) @db.Uuid // 用户UID 用来reset password
    name      String // 名字
    ...
}

数据库的连接串是从环境变量DATABASE_URL拿到. 每次修改schema.prisma，都需要运行prisma migrate dev生成迁移脚本，同时运行prisma generate可以生成基于TS的客户端@prisma/client来直接使用。

import { PrismaClient } from '@prisma/client'
const prisma = new PrismaClient()
// use `prisma` in your application to read and write data in your DB*
await this.prisma.user.findMany({...})

Prisma Concept Generating the client

Prisma上Dockerfile

接下来需要将该应用打包为Docker镜像，部署到Kuberntes的集群里。一开始的Dockerfie是这样的：

FROM node:16-alpine

WORKDIR /home/node
COPY . /home/node

RUN yarn install
RUN yarn prisma generate

ENV NODE_ENV production
RUN yarn run build-frontend
EXPOSE 7021

CMD ["node", "dist/apps/frontend/main.js"]

第一条指令是选定了合适的基础镜像包：基于Alpine Linux轻量级操作系统镜像上NodeJS v16版本。这块可以看看https://hub.docker.com/_/node，几乎每个版本都有默认，bullseye和alpine等等版本，对应的每个镜像包含的功能和大小也是不一样的，可以进一步到github查看其原始的Dockfile定义. 后续的指令就是将本地源文件都复制到镜像中，并运行yarn install安装依赖，prisma generate生成@prisma/client， nest build生成dist，最后“node”, “dist/apps/frontend/main.js运行整个程序。

Prisma Generate报错

当运行docker build -t frontend-api -f ./Dockerfile.frontend .来构建这个镜像时候，会在RUN yarn prisma generate报如下错误：

#9 3.978 Prisma schema loaded from libs/db/prisma/schema.prisma
#9 4.849 Error: Unable to require(*/home/node/node_modules/prisma/libquery_engine-linux-musl.so.node*)
#9 4.849  Error loading shared library libssl.so.1.1: No such file or directory (needed by /home/node/node_modules/prisma/libquery_engine-linux-musl.so.node)

看起来这里有两点：第一点无法加载 libssl.so.1.1这个共享库，第二点这个库是被node_modules/prisma/libquery_engine-linux-musl.so.node所依赖。

简单的从名字可以推测出来是关于SSL的，有可能是OpenSSL 1.1这个库没有安装，所以无法引用到。进一步搜索libssl.so.1.1可发现Openssl can’t find libssl.so.1.1 and libcrypto.so.1.1 · Issue #19497 · openssl/openssl (github.com)和Prisma官方的github issues Support OpenSSL 3.0 for Alpine Linux · Issue #16553 · prisma/prisma (github.com) 中其他人类似的问题反馈可以验证之前的猜想。沿着这个基础镜像往上推导，可以找出到底这个基础镜像包含哪些基础的依赖或者组件库，是否有OpenSSL还是说OpenSSL有但是版本不对等等，从dockerhub的Node镜像地址直接进入到Github对应的Dockfile定义docker-node/16/alpine3.15/Dockerfile:

FROM alpine:3.15
ENV NODE_VERSION 16.13.1
...
  && apk add --no-cache --virtual .build-deps-full \
        binutils-gold \
        g++ \
        gcc \
        gnupg \
        libgcc \
        linux-headers \
        make \
        python3 \
  && yarn --version

这里可以看到具体node-16:alpine是基于哪个NodeJS的版本和哪个Linux Alpine的发布: 16.13.1和3.15，后续当docker run起来之后可以cat /etc/os-release进一步确认。下面接着安装NodeJS和基础的开发依赖，里面是没有openssl这个库的。这个问题就变成了如何在prisma generate之前安装这个openssl1.1这个组件，这就很简单了，使用APK安装即可：

FROM node:16-alpine
WORKDIR /home/node
COPY . /home/node

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories #国内换成阿里
RUN apk update  #更新apk来源，保证可以拉倒最新的包信息
RUN apk add openssl1.1-compat #安装openssl 1.1

重新docker build一下，构建成功！

Prisma Generate原理

虽然目前这个问题看起来是解决了，但是还有一个疑问这个node_modules/prisma/libquery_engine-linux-musl.so.node文件-引用SSL的-prisma的组件是干嘛用的，为什么需要引用，另外跟prisma generate有什么关系，这些是需要接下来搞清楚的。

查看官方文档说明Generating the client (Concepts) (prisma.io)，当运行prisma generate时候，prisma会在node_modules/.prisma/client下面生成三个组件：

ts类型定义 (index.d.ts)
JS代码 (index.js)
查询引擎二进制文件（libquery_engine-xxx.xx.node）

下图是在本机MAC上的截图：

回忆在当初安装yarn add @prisma/client这个包时. @prisma/client包其实有两部分组成：

* The @prisma/client module itself, which only changes when you re-install the package:  *node_modules/@prisma/client* (重新安装包时才变化)
* The .prisma/client folder, temporary, which is the default location for the unique Prisma Client generated from your schema: *node_modules/.prisma/client* （临时只有运行prisma generate就会重新生成）

import { PrismaClient } from '@prisma/client'
const prisma = new PrismaClient()
// use `prisma` in your application to read and write data in your DB*
await this.prisma.user.findMany({...})

在调用this.prisma.user.findMany时，Prisma Client客户端将findMany发送给查询引擎（NodeJS-API Library)，查询引擎将其翻译为SQL语句，然后发送给数据库；当数据库返回结果时，查询引擎将其翻译映射为JS对象，并发送回给Prisma Client客户端。

Prisma Concept Generating the client

为了保证更高的效率，这个查询引擎针对每个不同的操作系统都做了相应的优化，都有相对应的编译出来的二进制文件。它命名的格式一般是 query-engine-PLATFORM或者libquery_engine-PLATFORM，这个PLATFORM指代不同的平台。比如我的电脑是macOS Intel，操作系统是达尔文Darwin，那么对应的查询引擎的名字是libquery_engine-darwin.dylib.node，如果上图所示。在上面的Dockerfile里，操作系统是Alpine3.15，那么其对应的名称应该是什么了？在Prisma schema API (Reference)，可以找到是，名字应该是libquery_engine-linux-musl.so.node:

名称是linux-musl, 对应的要求依赖的openssl版本是1.1.x，所以上面才需要在From node-16:alpine指定之后安装openssl 1.1版本。但是在Prisma4.8.0之后，大概是2022十二月发布的版本，优化改进支持了OpenSSL 3.0: Support OpenSSL 3.0 for Alpine Linu.

在CI上构建拉取libquery_engine-linux-musl.so.node的时候，有时候比较慢，可以加上ENV PRISMA_BINARIES_MIRROR http://prisma-builds.s3-eu-west-1.amazonaws.com来加速下载。

安装 OpenSSL 1.1 -> 下载该操作系统上的查询引擎二进制 -> prisma generate

如何数据迁移Prisma Migrate Deploy

接下来问题是如何跑迁移脚本： Deploying database changes with Prisma Migrate 里提到要保证./libs/db/prisma/migrations文件夹存在然后在发布阶段跑prisma migrate deploy - 命令来自@prisma/client包，而不建议在本地跑远程数据库的迁移。这里需要将package.json中的@prisma/client从devDependencies开发依赖挪到dependencies产品依赖，保证不会被CI或者部署平台(类似Vercel)prune修剪掉开发依赖导致无法运行命令。

- CMD ["node", "dist/apps/frontend/main.js"]
+ ENTRYPOINT [ "npm" ,"run"]
+ CMD ["start:prod_frontend"]

package.json:

  "scripts": {
    "build-frontend": "nest build frontend",
    "start:prod_frontend": "yarn prisma migrate deploy && node dist/apps/frontend/main",
  }
  "prisma": {
     "schema": "libs/db/prisma/schema.prisma"
   }

Dockerfile优化

上一步将Prisma成功的集成到了Dockfile里，并且部署之后容器成功的跑了起来，看起来流程没有问题。但是这个打包的镜像非常大，有1.3G，并且构建时间非常长，这相当不利于CI/CD的快速迭代演化。

server ➤ docker images                                                                                                                                                                                                                                              REPOSITORY     TAG       IMAGE ID       CREATED             SIZE
frontend-api   latest    93727cf78c85   About an hour ago   1.53GB

在怎么优化之前，可以先了解Docker如何根据Dockerfile里的指令构建镜像的。

FROM node:16-alpine  #Layer n
USER root            #Layer n+1
WORKDIR /home/node   #Layer n+2
COPY . /home/node    #Layer n+3
RUN yarn install     #Layer n+4
RUN apk update       #Layer n+5
RUN apk add openssl1.1-compat #Layer n+6
RUN yarn prisma:generate      #Layer n+7
ENV NODE_ENV production       #Layer n+8
RUN yarn run build-frontend   #Layer n+9
EXPOSE 7021                   #Layer n+10

ENTRYPOINT [ "npm" ,"run"]    #Layer n+11
CMD ["start:prod_frontend"]   #Layer n+12

这个过程每一行指令就像堆积木一样不断在前面的层上面添加新的层。比如上面的基础层是Layer n 就是From node:16-alpine（本身内部也是由其他层堆叠起来），运到后面的指令比如COPY . /home/node表示将当前本地文件夹（context)的文件复制到镜像的新的一层Layer n+3, 位于Layer n+2之后。RUN这里主要用来安装一些系统依赖或者组件。这里重点关注在COPY和RUN这两个指令，一个是文件系统操作，一个是bash或者命令操作。最终所有Dockfiler指令跑完之后，就得到了一个由很多层堆叠起来的镜像，镜像的大小就等于这些层的总和。

这个过程很耗时，当你需要不断的重复性的构建时，比如本地开发调试时，这就很痛苦了。所以Docker提供了构建缓存来加快构建过程。回到上文中的堆积木的这个比喻，每当有一层积木发生变化，后面的积木层都需要重新搭建。当比如构建上下文下的源文件发生变化比如./libs/db/prisma/schema.prisma发生改动，那么COPY . /home/node这个指令会检测到变化，通知下面所有的层都需要重新构建，也就是说Docker会令该层的缓存失效。相反，如果那一层没有变化，那么将会直接使用之前的缓存，这样构建速度就会加快。

分析下上面的Dockerfile的写法，可以知道它不是很有效率。如果优化了，有几个方向可以试试：

1.更好的组织构建顺序

COPY . /home/node将会将所有文件到镜像层中，后面紧接着会安装项目依赖yarn install。按照上面所说，当修改了跟依赖管理无关的代码（非package.json和yarn.lock)也会触发该yarn install重新安装所有依赖，即使上次之后依赖并没有发生变化。实际上这里COPY干了两件事情，一个是跟依赖相关的，一个是源代码。两个修改的频率频次也是各不相同，修改项目依赖的频次远远低于修改源代码的频次。所以可以拆成：

COPY package.json yarn.lock .yarnrc .
RUN yarn install #先安装项目依赖
COPY . . #复制源代码
RUN apk update
RUN apk add openssl1.1-compat
RUN yarn prisma:generate
ENV NODE_ENV production
RUN yarn run build-frontend

这里给了一个如何做分拆的角度：频率频次。大多数项目都可以分为：

准备系统，基础镜像 – 一次行为
安装系统依赖 – 一次行为
安装项目的依赖 – 频率略高，当你修改了项目的依赖 package.json/requirements.txt等等
修改源代码 – 频率最高，功能开发
构建
运行

按照这个顺序来的话，Dockfile可以这么调整：

# 1.初始基础镜像
FROM node:16-alpine
USER root
# 2.安装系统依赖
RUN apk update
RUN apk add openssl1.1-compat
# 3.安装项目依赖
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc /home/node
RUN yarn install

# 4.复制源代码，生成项目Prisma
COPY . .
RUN yarn prisma:generate

# 5. 构建
ENV NODE_ENV production
RUN yarn run build-frontend

# 6. 运行
EXPOSE 7021
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

2.减少复制到镜像每一层的文件大小

COPY . .这个命令需要谨慎使用，需要思考复制这些文件是要干嘛，为什么而使用。虽然你可以在.dockerignore里声明一些复制时需要忽略的文件，但是有些时候黑名单的方式有点不方便。可以在.dockerignore里声明一些通用性的忽略规则，比如node_modules,.env和NestJS这边的dist目录。显示声明的好处在于迫使你思考你需要这些文件是用来干嘛的，从而避免一些不必要的缓存失效从而影响构建时间和效率。

上面Dockerfile里面的第四步：

# 4.复制源代码，生成项目Prisma
COPY . .
RUN yarn prisma:generate

这个COPY文件的目的在于为了后面的yarn prisma generate来生成prisma的客户端。从上面关于Prisma工作机制我们知道它只需要有package.json里声明的prisma>schema的值也就是libs/db/prisma/schema.prisma文件就足够能完成这个prisma客户端的生成。所以这里如果修改了非libs/db/prisma/schema.prisma的源代码比如main.ts等等也会触发这个缓存失效，重新运行yarn prisma generate，这个就不是很有效率。

同时我们知道构建时候我们只需要apps/frontend部分，因为apps/backend和apps/frontend-emp都跟我们此次构建的目的没有关系。同时复制nest build需要的一些配置参数文件和公共组件libs目录，COPY 可以接受多个参数。

# 4.生成项目Prisma
COPY libs/db/prisma/schema.prisma ./libs/db/prisma/schema.prisma
RUN yarn prisma:generate

# 5. 构建
COPY apps/frontend ./apps/frontend #复制nestjs下面前端模块源代码
COPY libs ./libs #公共模块libs文件夹
COPY nest-cli.json tsconfig.json libs .  #nest build相关需要的配置
ENV NODE_ENV production
RUN yarn run build-frontend

3. RUN合并命令

在计算是否命中缓存时候，官方文档Best practices for writing Dockerfiles: Leverage build cache中提到，Docker对于COPY命令式会读取文件内容做一个checksum校验和，跟镜像层理的文件校验和checksum做对比。但是对于命令行类型的指令RUN而言，它不看命令产生的变化是否相同，而是简单的对比字面上两个命令字符串是否一样。

就RUN指令常见的应用apt-get和apk命令

# 2.安装系统依赖
RUN apk update
RUN apk add openssl1.1-compat

这个看起来没什么问题。apk update获取最新的安装源信息，然后apk add安装到最新的版本。当它第一次build的时候，apk update拉取最新的信息，然后安装了openssl1.1-compat的最新版本假设是1.1.0，假设第二次build的时候，夸张点，是十年之后，RUN后面的apk update命令跟之前是一样的，所以Docker不会去拉取最新的安装源，apk add openssl1.1-compat 跟之前也一样，所以不会重新安装，还是沿用之前的1.1.0。

APK repo:

openssl1.1-compat=1.1.0 wget=1.0 curl=1.0

十年之后需要安装另外一个系统依赖库，比如wget，于是这么写：

# 2.安装系统依赖
RUN apk update
RUN apk add openssl1.1-compat wget

十年之后的APK repo:

openssl1.1-compat=1.1.9 wget=1.9 curl=1.9

这个时候就有意思了： apk update 这一层不会重新构建，还是因为命令字面上没有变化，apk add openssl1.1-compat wget却跟已有镜像层里的apk add openssl1.1-compat字符串不一样，于是重新执行安装了最新的版本的wget, 但是openssl1.1-compat的版本是1.1.0, 而wget的版本却是十年前的老版本 :)

所以一般来说会将RUN的指令合并：

- RUN apk update && apk add openssl1.1-compat
+ RUN apk update && apk add openssl1.1-compat wget

这样一来重新构建的时候，Docker知道RUN这个命令有变化，会重新跑一遍，拉取最新的安装源信息，并安装最新版本，这个叫cache busting. 但是紧接着还有一个问题，wget是拉取安装了最新版本了，但是此时openssl1.1-compat如果安装源有更新比如版本到了1.1.9，也会重新安装，此时可能就不是当时的1.1.0, 而这样可能会带来意想不到的问题。

所以最好是在安装时候确定依赖的具体版本version pinning，用xxx=1.x.x指定特定的具体的版本，而不是一个大小版本号区间。APK这里可以去Alpine Linux packages查询对应的库的版本有哪些，比如 openssl1.1-compat的在构架下版本是1.1.1t-r0，那么Dockfiler可以更新为：

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add openssl1.1-compat=1.1.1t-r0

正如Docker在构建时会使用缓存来提高重复构建的效率一样，系统级别的包管理工具apk/yum/apt-get和后面要提到的NodeJS的包管理工具npm/yarn，也是类似的缓存来提高包安装效率。简单的说，在这里APK在安装时候会将安装包在 /var/cache/apk/目录下进行缓存，当你下一次需要安装某个包的某个版本时候，它会检查缓存中是否已经有了，如果有，那么直接使用，这样就加快了包安装的速度。

在安装过程中，产生了中间产物，也就是缓存，这个需要在结束之后清理的掉：

RUN  apk update && apk add openssl1.1-compat=1.1.1t-r0 && rm -rf /var/cache/apk/*

或者使用apk add –no-cache openssl1.1-compat=1.1.1t-r0来禁止安装过程中使用缓存，这样就不会产生额外的中间产物。

鉴于系统依赖的更新频率不高的特点，对于在不同构建过程中的缓存的使用，甚至是多个不同项目的构建，上面这个其实足够了。但是如果你经常需要改动，类似于下面要提到的项目依赖那么高的频次，那么可以考虑使用BuildKit特性中RUN –mount type=cache:

RUN \
    --mount=type=cache,target=/var/cache/apk \
    apk update && apk add openssl1.1-compat=1.1.1t-r0

这样的话/var/cache/apk/目录下内容会在不同的构建之间得到保存，而不是随着构建完成而结束，就好比外挂载了一个专门的数据卷Volume. 当apk需要重装时候，就可以利用apk的缓存/var/cache/apk，减少无谓的网络请求和消耗，提高构建速度。这块也会在后面的yarn/npm里讲到。

Note：对于每个Dockerfile指令，需要什么输入，产生什么输出，有什么中间产物，做什么事情，就跟面向对象里面向接口编程一样，合同contract是什么，必要时需要了解它是怎么工作的

4. npm/yarn安装依赖

在构建的时候RUN yarn install非常慢，而且在整个1.53G的镜像中，它这一层占了1.19G这么大，所以这个肯定是优化的一个大头。Yarn安装时候会将所需的包从npm registry下载下来，缓存到YARN_CACHE_FOLDER指定的目录下，比如mac上是yarn cache dir = /Users/tuo/Library/Caches/Yarn/v6，这里缓存的信息包括压缩包和元数据（时间戳版本Etag等等）, 然后将压缩包解压到项目下面的node_modules对应的目录下面。

可以想象每次构建都会重复这个流程，解析package.json和yarn.lock, 然后去远程下载包和元数据Package Metadata到缓存目录（在镜像层里），然后安装到node_modules目录下。每次构建都是一次性的，如果有改动，比如新加了一个包，那么这个流程需要重新再来一次，可以想象这个速度肯定是很慢的，这个缓存甚至有点多余，简直是个累赘。还产生了多余中间产物，就是/usr/local/share/.cache/yarn/v6目录下。

RUN yarn cache dir && yarn install && yarn cache clean

虽然改动项目依赖的频次可能没有改源代码的频次那么高，但是实践来看也不小，每次都得这么来一遍就比较折磨人的了。如果可以将这个缓存中间产物从构建镜像的流程中单独拎出来，达到类似数据卷的一样的目的，用的时候挂载上去，用完了卸载下来，这样一来便可以加快构建流程。当新增一个包，其他的包都可以从缓存里读取出来，只需要从远程拉取新增的这个包到缓存并放到对应的node_modules下面即可。官方应该看到了这个常见的需要，提供了新的构建器BuildKit (docker.com)，相对老版本的构建器，增加了很多性能方面的优化，不仅仅是我们这里提到的RUN –mount type=cache.

RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install

这里指定了target是/cache/yarn目录，同时设置YARN_CACHE_FOLDER为此目录来设置yarn安装时候缓存的路径。默认新版的Mac版本的Docker Desktop是支持并启用Buildkit的，无需配置；如果是其他系统，需要检查下docker的版本。

<missing>      2 minutes ago    RUN /bin/sh -c YARN_CACHE_FOLDER=/cache/yarn…   628MB     buildkit.dockerfile.v0

这一步之后，这一层的体积从之前1.19G下降到了628M，还不错。

接下来模拟新加一个包，yarn add underscore@1.13.6 然后构建一下：

=> [stage-0 5/9] RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install                                                                                                                                                                   114.3s

大概是114s 两分钟，这个时长也太长了点吧，不是缓存了所有其他的包，就只需要安装这个understore包么？需要接近2分钟？

这个时候我们需要简单了解下yarn install的原理，默认来说每次安装会根据version,name和integrity去缓存查找是否对应的版本.yarn-metadata.json和.yarn-tarball.tgz;

如果存在，还会发送一个304检查的请求，查看该本地缓存中版本的信息是否过期；如果过期，那么使用新的数据刷新缓存，否则直接使用缓存中的数据；
如果不存在，直接从远端拉取数据到缓存

flowchart TD a0[["yarn install"]] a1{"has internet connection?\n是否联网？"} a2{"matched in local cache?\n本地缓存是否命中?"} a3{"remote check if got expired?\n远程访问校验是否过期?"} a4{"fetch \nand update cache \nthen unzip to node_modules\n拉取更新本地缓存\n并解压到node_modules"} a10["use cache\n使用本地缓存"] a20["fetch and update cache\n远程拉取并跟新本地缓存"] a30["use cache\n使用本地缓存"] a0 --> a1-->|YES|a2-->|YES|a3-->|YES|a4 a1-->|NO|a10 a2-->|NO|a20 a3-->|NO|a30

所以这个额外的304检查开销应该就是导致缓慢的原因，特别是你的包越多，那么请求的次数和总的时间就多。但是因为用来yarn.lock来精确锁定版本号，是不是可以跳过这个304请求了？

从这两篇文章NPM v5.0.0 prefer-offline和 PNPM prefer-offline的说明来看，--prefer-offline可以来设置缓存策略为离线优先，直接匹配缓存中的数据，如果有直接使用，而不用发额外的过期检查请求。这样只有当包在缓存中没有时，才会发生网络请求，这样理论上应该会快很多。尝试安装另外一个版本yarn add underscore@1.13.2

=> [stage-0 5/9] RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install --prefer-offline                                                                                                                                                   61.7s

时间从114.3s降到了61.7s，效果还可以 :)

如果CI/CD构建是完全离线的，你甚至可以使用官方Running Yarn offline中那样，配置yarn-offline-mirror目录（这个跟yarn cache是不一样的）和yarn-offline-mirror-pruning将依赖包本地生成并提交到代码仓库了，这样在构建时候，可以直接从yarn-offline-mirror目录读取缓存的离线安装包直接安装。

5. 使用MultiStage多阶段

实际目前打包出来的镜像还是很大，yarn install这块还是占了很多空间。这个时候需要跳出来想想最终输出的产物是什么。Nest build时候会生成dist目录，但是不像其他的语言GoLang等，安装依赖+源代码，编译之后产生一个exe等二进制文件直接丢出去运行即可，这里还需要node_modules目录，包含第三方的依赖。所以出来的是两个目录，一个是dist，一个node_modules。这里将yarn install 需要安装devDepencies和depencies成为dev-dep环节，只需要安装产品发布的depencies传给成为prod-dep环节, 那么就有如下关系：

node_moduels/.prisma/client的生成来自prisma generate - 需要dev-dep环节之后, 跟随libs/db/prisma/schema.prisma发生改变
dist目录的生成nest build需要dev-dep环节之后（nest-cli是dev依赖），确切的说第一步之后，因为源代码有引用PrismaClient，跟随源代码发生改变
node_modules最终是第一步生成node_moduels/.prisma/client 加上 prod-dep环节下的node_modules

简单的梳理了分为四个阶段，而且是线性的，每个阶段的产物可以被下一个阶段使用。

FROM node:16-alpine as dev-dep
USER root
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc .
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --prefer-offline

FROM node:16-alpine as prisma-binary
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache openssl1.1-compat=1.1.1t-r0
WORKDIR /home/node
COPY --from=dev-dep /home/node/node_modules/ ./node_modules/
COPY package.json yarn.lock .yarnrc .
COPY libs/db/prisma/schema.prisma ./libs/db/prisma/schema.prisma
RUN yarn prisma:generate

FROM node:16-alpine as nest-build
WORKDIR /home/node
COPY --from=prisma-binary /home/node/node_modules/ ./node_modules/
COPY . .
RUN yarn run build-frontend

FROM node:16-alpine
ENV NODE_ENV production
WORKDIR /home/node
COPY  --from=prisma-binary /home/node/node_modules/.prisma ./node_modules/.prisma
COPY  --from=nest-build /home/node/dist ./dist
COPY package.json yarn.lock .yarnrc .
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --prefer-offline --production
EXPOSE 7021
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

这里利用了Multi-stage根据不同的目的来划分为不同的阶段，好处在于当你操作COPY/RUN等命令时不用担心需要清楚中间产物，可能需要一些shell脚本来清除中间产物，保证镜像层不打包无用的文件。一般来说都有完整依赖的开发环境和裁剪瘦身过后的生产环境两个阶段，这里极端的用了四个阶段。得到的最终文件输出去：

frontend-api latest 271517ddc574 42 seconds ago 656MB

镜像大小不降反升！

其实是可以优化下阶段组成的，一个原因是都是线性的，另外一个在不同阶段中间进行copy也是比较费时的。于是回到之前的Dockerfile，改成两个阶段：

# 1.初始基础镜像
FROM node:16-alpine as dev-dep
USER root
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache openssl1.1-compat=1.1.1t-r0

WORKDIR /home/node
COPY package.json yarn.lock .yarnrc ./
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --prefer-offline

COPY libs/db/prisma/schema.prisma ./libs/db/prisma/schema.prisma
RUN yarn prisma:generate
COPY . .
RUN yarn run build-frontend

FROM node:16-alpine
ENV NODE_ENV production
WORKDIR /home/node
COPY  --from=dev-dep /home/node/node_modules/.prisma ./node_modules/.prisma
COPY  --from=dev-dep /home/node/dist ./dist
COPY  --from=dev-dep package.json yarn.lock .yarnrc ./
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --production --ignore-scripts --prefer-offline

# 6. 运行
EXPOSE 7021
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

镜像包大小从656MB下降到了567MB。这里有一个问题，当改了源代码之后，yarn install –production就会重新跑一遍，这个是多余的。而且有一个问题就是yarn install –production并不会删除devDependencies的那些依赖，也并没有像npm一样有npm prune来裁剪那个dev依赖，只能是借助于一些自定脚本或者别的方式，npm prune equivalent behavior · Issue #696 · yarnpkg/yarn (github.com) 整体而言都不简单。但是我们可以借助于multistage来绕过这个问题。所以整理下，新的应该是这样的：

Dev-dep - yarn install , prisma generate, nest build
prod-dep

最后合起来。

flowchart TD A[base] --> B{yarn install\nprisma generate\nnest build} A[base] --> C{yarn install --production} B --> |node_module/.prisma/client, dist|D{npm run start} C --> |node_module|D[npm run start]

Dockerfile是这样的:

FROM node:16-alpine as dev-dep
USER root
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache openssl1.1-compat=1.1.1t-r0

WORKDIR /home/node
COPY package.json yarn.lock .yarnrc ./
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --prefer-offline

COPY libs/db/prisma/schema.prisma ./libs/db/prisma/schema.prisma
RUN yarn prisma:generate
COPY apps/frontend ./apps/frontend
COPY libs ./libs
COPY nest-cli.json tsconfig.json libs .
RUN yarn run build-frontend

FROM node:16-alpine as prod-dep
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc ./
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --production --ignore-scripts --prefer-offline

FROM node:16-alpine
ENV NODE_ENV production
WORKDIR /home/node
COPY  --from=prod-dep home/node/node_modules/ ./node_modules
COPY  --from=dev-dep /home/node/dist ./dist
COPY  --from=dev-dep /home/node/node_modules/.prisma ./node_modules/.prisma
# need migrations
COPY libs/db/prisma/ ./libs/db/prisma/
COPY package.json yarn.lock .yarnrc ./

EXPOSE 7021
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

运行一下得到的大小是419MB。

7 minutes ago   COPY /home/node/node_modules/.prisma ./node_…   81.7MB    buildkit.dockerfile.v0
7 minutes ago   COPY home/node/node_modules/ ./node_modules …   218MB     buildkit.dockerfile.v0

可以看到除了基础镜像外，已经将node_modules优化到了218MB， .prisma/client到了81.7MB.

这是我们优化的历史：

6. Dockerfile用户权限

默认来说Docker容器是以root用户的运行的，这个root用户跟宿主机上的root用户是一样的，UID为0。这样一来就意味着如果攻击破解了容器，就可以直接操作宿主机，安全风险很大。既然默认用户是root，那么本着Docker指令能少就少的原则，第一个阶段里的指令USER root就可以删掉了。官方建议如果运行非必需ROOT权限，尽量保持权限最小，切换为非root的用户。添加用户群组的时候，显示的指定UID/GID，就像node-16:apline的dockerfile中创建node用户一样：

FROM alpine:3.17

ENV NODE_VERSION 18.14.2

RUN addgroup -g 1000 node
&& adduser -u 1000 -G node -s /bin/sh -D node

这个时候/home/node目录的权限就是node用户下面。如果你需要切换不同权限的用户，不建议使用sudo，因为其机制问题：一个是会启动2个进程（父子），第二个是信号传递和TTY的问题, 可以考虑使用tianon/gosu: Simple Go-based setuid+setgid+setgroups+exec (github.com).

要设置运行容器的时候，可以在Entrypoint和cmd之前，设置需要的用户，当然这个用户必须是声明的过的。

EXPOSE 7021
USER node
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

然后docker build 并运行出现如下报错：

@tuo.local ➜ server rvm:(system) git:(uatFt) ✗ docker run 09ba84abb858
$ /home/node/node_modules/.bin/prisma migrate deploy
Error: Can't write to /home/node/node_modules/@prisma/engines please make sure you install "prisma" with the right permissions.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

貌似是写入node_modules权限不够，因为yarn install是root，但是当运行prisma migrate deploy的用户是node，没有足够的权限写入。

为了验证想法，我们最好可以直接进入容器内部的shell(docker exec -it xxx /bin/sh)，直接ls查看文件夹的权限，但是此时容器刚启动就因为权限问题退出了。这里可以在最后一步安装bash的库，同时将Entrypoint和CMD改成tail -f让容器一直保持运行。

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache bash
EXPOSE 7021
USER node
#ENTRYPOINT [ "npm" ,"run"]
#CMD ["start:prod_frontend"]
#dev purpose
ENTRYPOINT ["tail", "-f", "/dev/null"]

运行docker exec -it 916c42e68c3b /bin/bash，可以进入shell：

916c42e68c3b:~$ ls -al
-rw-r--r--    1 root     root           430 Feb 24 10:40 .yarnrc
drwxr-sr-x    3 root     node          4096 Feb 25 15:19 dist
drwxr-sr-x    3 root     node          4096 Feb 26 03:53 libs
drwxr-sr-x    1 root     node          4096 Feb 25 15:19 node_modules
-rwxr-xr-x    1 root     root          8230 Feb 26 03:13 package.json
-rw-r--r--    1 root     root        376897 Feb 25 14:05 yarn.lock

916c42e68c3b:~$ ls -al node_modules/@prisma/engines
-rw-r--r--    1 root     node           537 Feb 25 12:40 README.md
drwxr-sr-x    2 root     node          4096 Feb 25 15:18 dist
drwxr-sr-x    2 root     node          4096 Feb 25 15:18 download
-rw-r--r--    1 root     node          1384 Feb 25 12:40 package.json

node用户对于node_modules/@prisma/engines只有读权限，自然无法写入。这个时候别急着chmod -R g=rwx ./node_modules/@prisma/engines，最好的办法是利用dev entrypoint+bin/bash直接登录上去在容器里模拟现实场景运行yarn prisma migrate deploy，可以发现是同样的错误。

但是问题来了，为什么需要往node_modules/@prisma/engines目录下面去写入了，而不是去生成的目录node_modules/.prisma/client写入？跟本地一对比可以看到在node_modules/@prisma/engines下面缺失了libquery_engine-linux-musl.so.node这个查询引擎二进制文件。虽然官方文档没仔细写，大概可以推测因为二进制文件是比较大，生成费时费力，而prisma generate或者prisma migrate deploy会跑的次数比较多，所以临时生成的文件夹（带有js，ts和binary target)，需要不断的生成，但是二进制文件基本上系统级别的依赖，不用每次跟着ts/js生成，所以放一份放到了@prisma/engines下面，这样假设.prisma目录删除重新生成时候，只需要从@prisma那里复制下二进制文件即可。所以当prisma检测到@prisma/engines没有二进制文件，就会主动去下载对应的查询引擎二进制文件，导致需要node_modules/@prisma/engines目录下写入。

所以解决办法就是在从devDep构建阶段那边复制.prisma依赖的同时复制@prisma，再一个因为openssl1.1并没有在Alpine里原生支持，所以还需要安装openssl1.1.

COPY  --from=dev-dep /home/node/node_modules/.prisma ./node_modules/.prisma
COPY  --from=dev-dep /home/node/node_modules/@prisma ./node_modules/@prisma  #同时复制
# need migrations
COPY libs/db/prisma ./libs/db/prisma
COPY package.json yarn.lock .yarnrc ./

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache openssl1.1-compat=1.1.1t-r0
EXPOSE 7021
USER node
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

构建之后发现运行没有问题：

docker run d61f5e95f91754a3ad8d71cf232a6f09a5a81025aa767b6a8170158bc132165b
yarn -v && printenv && yarn prisma migrate deploy && node dist/apps/frontend/main

$ /home/node/node_modules/.bin/prisma migrate deploy
Prisma schema loaded from libs/db/prisma/schema.prisma
Datasource "db": PostgreSQL database "simeProdFeb21bk1", schema "public" at "host.docker.internal:5432"

263 migrations found in prisma/migrations

No pending migrations to apply.
Done in 1.88s.
[Nest] 17   - 02/26/2023, 4:49:59 AM   [NestFactory] Starting Nest application...

这个方法是提前将二进制文件下载生成好打包到镜像中，好处是启动时候yarn prisma migrate deploy无法网络请求查询引擎二进制文件，坏处是生成的镜像包体积会变大640MB（相比之前420MB）。也就说明这个二进制文件其实还是挺大的，特别是两个地方@prisma和.prisma都有独立的文件，不是软链接。这个其实也有办法可以优化二进制文件的大小（后面会谈到），因为有两处使用，优化好应该可以节省不少空间。

还有一种是不复制@prisma和.prisma到镜像中，利用yarn prisma migrate deploy会自动检查拉取二进制文件带@prisma/engine并生成对应的.prisma/client目录.

- COPY  --from=dev-dep /home/node/node_modules/.prisma ./node_modules/.prisma #没有必要
- COPY  --from=dev-dep /home/node/node_modules/@prisma ./node_modules/@prisma #没有必要
+ RUN chmod -R g=rwx ./node_modules/@prisma/engines #开放此文件夹权限，因为yarn install是root,node用户只能读
+ ENV PRISMA_BINARIES_MIRROR http://prisma-builds.s3-eu-west-1.amazonaws.com #国内这个速度还行，不然有的你等

这样的好处是打包镜像体积很小，440MB左右。但是缺点是容器启动时候需要： 1.下载二进制文件，2：运行prisma generate生成客户端。这两件事会影响容器启动的时间，可以的话可以考虑自己架个内部镜像。

其他改善

还有一些其他的改进的建议：

尽量使用最新的基础镜像 - 不管从性能还是安全性角度来看
缩减每一层的打包的大小；比如yarn install这里，可以考虑使用包管理辅助工具比如depcheck - npm (npmjs.com)来扫描代码中没有使用到的包，从而将其剔除出去来减少node_modules的大小
尽可能的将项目依赖的版本保持最新。可以使用一些工具npm-check-updates - npm (npmjs.com)来检查是否有版本的更新，比如这里就有提示 prisma ^3.1.1 → ^4.10.1。实际上在prisma后来发布的版本里，他们已经优化了OpenSSL的支持，同时大幅度缩小了二进制文件的大小。

Release 4.10.0 · prisma/prisma (github.com)

Smaller engine size used in Prisma CLI

In 4.8.0, we decreased the size of the engines by ~50%, which significantly impacted Prisma Client, especially in serverless environments.

In this release, we’ve reduced the size of Prisma CLI by removing the Introspection and Formatter engines. The introspection functionality is now served by the Migration Engine. A cross-platform Wasm module has entirely replaced the Formatter Engine. This reduces the overall installation size for Prisma CLI.

在4.8.0版本里将查询引擎的大小缩小了50%，更符合了轻量化上云的要求。

经过这些优化，将最终镜像的大小缩小到了360M!

看看每一层的大小，主要是看prisma和node_modules:

最终Dockerfile

FROM node:16-alpine as base
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache openssl=1.1.1t-r1

FROM base as dev-dep
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc ./
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --frozen-lockfile  --prefer-offline

COPY libs/db/prisma/schema.prisma ./libs/db/prisma/schema.prisma
RUN yarn prisma:generate
COPY apps/frontend ./apps/frontend
COPY libs ./libs
COPY nest-cli.json tsconfig.json libs .
RUN yarn run build-frontend

FROM node:16-alpine as prod-dep
WORKDIR /home/node
COPY package.json yarn.lock .yarnrc ./
RUN --mount=type=cache,target=/cache/yarn YARN_CACHE_FOLDER=/cache/yarn yarn install  --frozen-lockfile  --production --ignore-scripts --prefer-offline

FROM base
ENV NODE_ENV production
WORKDIR /home/node
COPY  --from=prod-dep home/node/node_modules/ ./node_modules
COPY  --from=dev-dep /home/node/dist ./dist
COPY  --from=dev-dep /home/node/node_modules/.prisma ./node_modules/.prisma
COPY  --from=dev-dep /home/node/node_modules/@prisma ./node_modules/@prisma
# need migrations
COPY libs/db/prisma ./libs/db/prisma
COPY package.json yarn.lock .yarnrc ./

EXPOSE 7021
USER node
ENTRYPOINT [ "npm" ,"run"]
CMD ["start:prod_frontend"]

#dev purpose
# RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && apk update && apk add  --no-cache bash
# ENTRYPOINT ["tail", "-f", "/dev/null"]

引用

Auto Carp Fishing Part 2 - Make it Smart and Non-Attended

2021-12-04T04:55:32+00:00

Let’s take a look at what’s the common bite alarms with its smartness of “tell you when you’ve got a bite”:

Simplest one is the left bottom one - the bells. The bottom right one is the alarm reciever. The middle one is the bite alarm base.

When a fish gets a bite and tugs at the fishing line, it would apply a force on the fishing line which makes the rod tip bend or vibrates. The simplest bite alarm is a fishing bell-like twin bells, just clamp to the rod tip. The upside is that the fishing bells are cheap, lightweight, no battery is needed; the downside is it is not that sensitive, also the sound is not that loud enough so you could only stay close to where that rod and bells are. Anyway, it does what it needs to do. Here in China, it is widely used. The advanced one is the bite alarm with alarm transmitter(base) and receiver(remote), with an adjustable volume, tone, and sensitivity. The alarm transmitter is like a bell which placed on the rod pod or rod rest kinda bells, however, the alarm receiver is a wireless companion device like a mobile phone which could pick up the signal from the alarm transmitter and alert you by vibrating, lighting up, or emitting a beep themselves with a range of some 50-150 meters(A decent bite alarm receiver like JRC Radar CX Receiver £53.99 = 450RMB, its range could be 150 meters). The range would depend on conditions, for example, how many obstacles are in the way. It gives you some freedom so that you could other things away from the rods like making a coffee and tidying up your barrow.

If you’re not worried about your rods/reels getting stolen and your home is like in 150 meters range to where you’re fishing, then after you could cast out rods and set up the bite alarm, you could just go back home and enjoy yourself.

That’s kinda the idea of what I want to do but it is not smart and stealthy enough. Here are my two cents:

Smartness.

The bite alarm works like a walkie-talkie (Two-way Radio). The transmitter - the alarm on the rod pod - sends a radio transmission which can be picked up by a receiver best through its antenna. The useful direct range of a two-way radio system depends on lots of radio propagation conditions, basically kinda limited. But I don’t want to have it limited by some distance like 150 meters, I just want it to be as freely used as how we use our smartphones. Also, that style of vibration or beep sound is just boring and archaic, how about some custom Ringtone from the phone in my pocket, e.g, knock on wood, something that every angler would wish they could say to the carps:

Hey, I just met you, and this is crazy. But here’s my number, so call me maybe.

Yeah, Call Me Maybe from Carly Rae Jepsen. vibe :) Just be smart enough to give me a call when it detects the bites.

Stealthiness.

Leave your eye-catching rods and reels, usually the most expensive fishing tackles, on the bank, is just inviting people to steal. It is best to remove those rods and reels when the cast is done by linking the end of the fishing line to the tent peg staking out on the ground. But this comes with a big challenge when fighting the fish, I will talk later. Also, the Smart Alarm’s components need to be well layout and fit compactly in a case that its size could be as small as possible, so that the whole device could be camouflaged like Rambo to not get distinguished by the passersby, let alone ones from some distance.

Bite Alarm Approaches

There are basically two ways when coming to how we setup the bite alarm, tent stakes with the fishing mainline:

onshore

offshore

We could put the bite alarm device on the bank or wrap it in a ball-shaped buoyant floater. They share the same idea the sensor part of the device could detect some gyroscope or accelerator changes: either like in the first case you put the senor on the inclined spring and once fish tug the line, the spring will be lifted up and senor could get that angle changes; or in the second case, the senor inside the float ball would detect a vertical acceleration（z-axis) when the fishing line is tugged from the bottom.

To make it more vividly, I did a little experiment to simulate a fish bite like following gif:

Both have pros and cons. The offshore approach’s pros:

a good stealthiness. People could not reach it easily over the water.
precision is higher given the shorter distance from float to the fish
not limited by onshore/bank conditions
serve as a bite indicator. No matter fishing coming towards the bank or running away from the bank, it always got trigged.

its cons:

pretty difficult to get it waterproof yet easy to open/close to setup device.
subject to weather conditions. if too windy or too much current, float ball - the big size and heavyweight - could be pushed back and forth, therefore causing the rigs and lead to moving away. Also if sunlight is strong enough it might get overheat inside the enclosed ball and cause electronics malfunction.
the casting process gonna be more complicated
gonna pretty much lose the device, fishing lines, and rigs when it got snagged on the bottom

I have tried a couple of pe or pvc products with sphere shapes like Gashapon and AB glues but the water resistance still remains a big headache. No existing product that suits my need could be found on taobao.

The onshore approach’s pros:

no concerns over the waterproofing, heating, etc
no change in the casting process, just normal casting
not limited by windy or other harsh weather conditions
just gonna lose the fishing lines and rigs when the hook got snagged or stuck in the rocks/weeds of the bottom. The bite alarm is intact.

its cons:

high demand on stealthiness. Still, people could possibly see it and just take it.
precision of bite signal is not that good given the long-distance fishing line.
the disguise requires a kinda blending with surroundings.

After weighing trade-offs between onshore and offshore approaches, onshore one seems to be a good start point as it is simpler and materially lower demanded. Next, let’s dive into the detailed mechanism.

Mechanism

sensor input: data collecting, detect the physical change of angle and acceleration and convert to digital/analog signal.
mcu coordinator: either poll the input senor or via interrupts to get data and check to see if it meets the criteria. If yes, send commands to an actuator output to perform some physical actions.
actuator output: establish a connection with a cellular base station nearby, register network. then take commands from MCU, send data over cellular data, and initiate a call to a certain mobile number.

Other than just sending phone-calling command when got a trigger, The MCU controller normally does some heartbeat check every few seconds or minutes by sending very few bits of data to some cloudplatform or backend server to log its health status.

Above is a typical scenario of the buzzword: Internet of Things (IoT). One of the unique things here, which is a good thing, is the communication between mcu and cloud platform is a single direction, i.e, only from MCU to cloudplatform, therefore no need to take any commands from cloudplatform to perform some actions to be autonomous.

A rule of thumb for IoT development is to always have its datasheet something like technical reference/documentation at hand. “More haste, less speed”, it is gonna save you lots of trouble and time. The datasheet is always something you could fall back on when you just have no idea or get into some clueless difficulties.

MCU - ESP8266

Consider the central processor/unit as the brain which is consisted of hardware and software. Based on this central unit there are microcontroller-based IoT boards like Arduino/ESP8266/STM32F and microprocessor-based boards like Raspberry Pi. But how much processing power it needs for our case? Not much. Surely Rasperry PI is overkill. Between the Arduino Uno and NodeMCU ESP8266, the esp8266 is the no-brainer. Arduino Uno board, which was used to be the dominant player in previous years, despite having a very mature and vibrant community, doesn’t have Wi-Fi capability built-in, has a voltage of operation of 5V, a physical size 69 mmx53 mm. Nowadays, NodeMCU ESP8266 board, which comes from a Chinese company Espressif, is the most popular one which comes with Wi-Fi built-in (extremely convenient to get started and play with sth), a voltage of operation of 3.3V(less power and current consumption mean power source could have more options hence cost are lower), a smaller size 58mmx31mm (easier to fit onto the breadboard and have a smaller overall case size to be more stealthy). Look at its price on Taobao:

NodeMCU ESP8266 vs Arudio Uno R3

Apart from that, ESP8266 could be programmed with C/C++ in Arduino IDE just like Arduino Uno. Even though generally bare-metal boards and chips don’t have an operation system, ESP8266 is one heck of IoT microcontroller - it does provide the capability to install some firmwares on the top of the Espressif Non-OS SDK for ESP8266 to abstract and simplify the development process.

Left picture source: #240 Time to Say Goodbye to Arduino and Go On to Micropython/ Adafruit Circuitpython? from Andreas Spiess.
On the right side you could see it took me 4.6 seconds to get led blinking code compile&upload in Arduino IDE

Every time you debug in Arduino IDE is kinda painful, you write in C/C++ code, compile, build and upload to board and this process could take some time even if you tuned some parameters like Tools -> Upload Speed to the max. But with that flashed firmware installed on the board first, you could just write the code in the language you’d like and save it then it got instantly executed on the board. Some language(tailored, not full set of feature) and its firmware for ESP8266 are: Lua based NodeMCU, Python based MicroPython, JavaScript based Mongoose OS. That should be the agile way of iot development :)

After installing the CH340 driver for mac, go to NodeMCU custom builds select modules(I have chosen those modules - bit, file, gpio, i2c, net, node, softuart, tmr, uart) to include for building the firmware. Then open NodeMCU PyFlasher to flash the downloaded bin file of firmware to the board.

Run the jar file of ESPlorer IDE to open the programming gui. The only problem with this on Mac is the serial port of the device sometimes doesn’t show up in the top right dropdown list, even though it is recognized by the system(“ls /dev/tty.*“). I have to go to settings and manually set the serial port of the device.

The entrance file of NodeMCU is the init.lua where we’re gonna import files from senor and actuator and code some trigger logic here.

Sensor - MPU-6050

The sensor module for detecting the angle and acceleration is the GY-521 MPU-6050 which is six-axis (Gyro + Accelerometer) + embedded temperature sensor motion tracking device.

We’re just gonna use 4 pins from MPU6050: sda, scl, gnd, vcc. Here is how I wire it with MCU Esp8266 and its schematics:

A good tip is to have your wire color coding schemes, for example a brighter color red/orange for vcc positive, a darker color black/blue for ground or earth.
more on: #12 Five Tricks for working with Dupont wires

You might wonder what does the sda and scl mean? Here is a very important topic - serial communication. To be as versatile as possible to communicate with all kinds of peripherals like sensors and actuators, it has a variety of forms of communication - it could be analog (voltage or current) or digital (using a protocol like I2C or SPI or UART). Each protocol has its requirements, limits, and use cases. What MPU-6050 communicates is a I2C(Inter-Integrated-Circuit) 2-Wire protocol. The two wires are serial data (SDA) and serial clock (SCL). The I2C device has some registered addresses for the system bus to look up. That’s something that got mentioned on its datasheet.

Chapter 4.32 in datasheet of mpu6050

Here is the code snippet for getting gyroscope in degrees/second unit, acceleration in g unit and termperature in degree/celcius. The main flow is setup resolutions like the unit you like to measure in, then inside a loop, retrieves its data and convert with proper scale factor or formula based on the configurations.

The mpu-6050 features a user-programmable gyro full-scale range of ±250, ±500, ±1000, and ±2000 °/sec (dps), and a user-programmable accelerometer full-scale range of ±2g, ±4g, ±8g, and ±16g, which could be found in its datasheet. You could get a more advanced one like Yaw/Pitch/Roll from arduino forum or someone uses it to control a drone. But the above one suffices in my case.

Actuator - SIM800C

In terms of connectivity, we need to take a look at the environment that we will be putting the device. There is no way it could have WIFI on the bank of the lake. Ideally, we want it could make a phone call and send network requests just like our mobile phones. A module that has the GSM for voice calling and sms, and the GPRS for cellular data transfer, would be a good fit.

The frequency of network requests is quite low - the heartbeat rate could be like every 1 minute. No privacy or security concerns here. An HTTP get/post request would work. And the data it needs to send is very small. No video streaming, no need for low latency, so the 3G,4G,5G is kinda overqualified, the best one here is the vintage 2G cellular network.

2G is kinda dying out in China. If you’d like you could have NB-IOT as an alternative. SIM7020C is a perfect substitute for SIM800C, with the same dimension and same layout.

SIM800C is a quad-band GSM/GPRS module in a LCC type that supports GPRS up to 85.6kbps data transfer. SIM800C has one SIM card interface. It integrates TCP/IP protocol.SIM800 can be controlled/configured using simple AT commands. MCU can send AT commands over the UART interface and control the SIM800. It can be used for sending/receiving messages, making calls, sending/receiving data over the internet, etc.

You could debug this module with AT commands(Here is the SIM800 Series_AT Command Manual_V1.12) via a USB-TTL converter to connect to your laptop. After connecting, using a wire to shorten the PWX pin on the GND on the sim800c board so that the module could start.

Pay attention the direction and the side of sim card when inserting into the slot

Then we could try AT commands in CoolTermMac on Mac, just choose serial port in Serial Port Options, choose Line Mode in Terminal Mode and switch from View Hex to View ASCII.

A couple of useful at commands to try out: AT+CPIN? check if sim card is ready; AT+COPS? to check which operator it is registered; AT+CREG? to request network registration status; AT+CSQ to show network signal quality; ATDXxxxxx; to call some phone number

The most important debugging tool for SIM800c is the network LED on the top right side of the SIM800C indicating the status of the cellular network. It has a different blinking rate. When powered up, the led will blink every 1 second to indicate that it is actively searching for a cellular base station therefore not connected to the cellular network yet. When it has made contact with the cellular network & can send/receive voice and SMS, all is good, it will blank every 3 seconds.

SIM800L is pretty much same to SIM800C. source: https://lastminuteengineers.com/sim800l-gsm-module-arduino-tutorial/

Connections and code snippets should be like below:

TX (SIM800C)-RX (D3)
RX (SIM800C)-TX (D2)
GND (SIM800C) - GND (MCU)
VBAT (SIM800C) - VCC (MCU)

Here is my code snippet, you could check out more on my github source repo.

---- Create new software UART with baudrate of 9600, D2 as Tx pin and D3 as Rx pin
function writeCMD(s, cmd)
    print("\n"..tostring(tmr.now())..": send cmd", cmd);    
    s:write(cmd.."\n");   
end

function sim_setup()
    if not s then 
        print("\n"..tostring(tmr.now())..": initilized su\n");
        s = softuart.setup(9600, 2, 3)
        s:on("data", "\n", function(data)
          local txt = string.gsub(data, "[\r\n]", "")
          print("\n"..tostring(tmr.now())..": receive from uart:", txt)            
        end)
    else
      print("\n"..tostring(tmr.now())..": existed su\n");  
    end
end

function sim_send(txt)
    writeCMD(s, 'AT+SAPBR=3,1,"Contype","GPRS"')
    writeCMD(s, 'AT+SAPBR=3,1,"APN","CMNET"')
    writeCMD(s, 'AT+SAPBR=1,1')
    writeCMD(s, 'AT+SAPBR=2,1')
    writeCMD(s, 'AT+HTTPINIT')
    writeCMD(s, 'AT+HTTPPARA="CID",1')
    local url = "http://xxxx.com/api/heartbeat?time="..tostring(tmr.now().."&txt="..txt)
    writeCMD(s, 'AT+HTTPPARA="URL","'..url..'"')
    writeCMD(s, 'AT+HTTPACTION=0')
    s:write(0x1a);  
end

function sim_call()   
    writeCMD(s, 'ATD186xxxx5235;')    
end

We send an http GET with txt parameter to some url. The txt is the string literal of content from MPU-6050 sensor. From the Nginx log of the backend server, we could see the following log:

As you see, when I put the MPU-6050 on the inclined spring and lift the spring up, from the log, I could see a sudden jump for ax, which I could tell whether something happened or not.

SIM800C Instability

There are lots of people in the forum or on youtube complaining about the stability issue of SIM800. They usually get some weird problems like the module keeps shutting down or the network led light blinks in a chaotic way and it won’t just function in a stable way. And I had this problem too, as you see, from above, how I wire and connect the module with MCU.

Power and current requirements and their consumption is often the most important part of IoT development. The NodeMCU - according to its datasheet, page 7, its operating voltage requirement is from 2.5v to 3.6v and inbuilt LDO voltage regulator of 3.3v - and the sensor, is powered by the one single battery of 18650 3.7v. The power source is another interesting and important topic also for IoT development and lots of factors need put into consideration. 18650 is based Lithium-Ion and is the most commonly used one and it is rechargeable. You could use an expensive one like Lipo based battery. Basically, you need to provide a voltage that is neither too high that is over the module’s upper limit nor too low like the battery keeps discharging and its voltage falls under that desirable voltage of the module making the module keep shutting down. For example, You could use 2 cells of 1.5v Alkaline non-rechargeable battery(1.5v * 2 = 3v), but it is like in the low side of the specification(2.5v-3.6v) and it would waste 50% of the energy of the battery, Or you could use 3 cells of AAA battery (1.5v * 3 = 4.5v), but 4.5v is way over the up limit of the specs (3.6v), therefore you need some LDO regulator to drop the voltage. So for a non-rechargeable battery, you kinda end up with some energy wasting or introducing some complexity. More on the battery supply options for ESP8266, there is a pretty good video from Andreas Spiess on <#64 What is the Ideal Battery Technology to Power 3.3V Devices like the ESP8266?>.

So it is always good to check its specs and datasheet.

Page 17 in chapter 4.1 Power Supply of sim800c datasheet

Most of time, the current consumption is not that much. But during the transmission burst, the current draw could reach to 2A. If the power supply couldn’t source 2A of surge current, its voltage will drop out and if it falls under 3V, the module will be shut down automatically.

The datasheet, suggests two approaches to work around this. (See, the datasheet not only just lists the specs but also provides solutions :))

5v LDO (low dropout regulator)

Use 5v power source like power bank etc, then use 2A-rated DC-DC Buck Converter like LM2596(a 3.0A Step-down switching regulator, 2.33RMB ~ 0.3$), connect it to VBAT pin on Sim800C.

3.7v Li-ion Battery (Recommended)

Li-ion battery like 18650 3.7v is perfect for the SIM800 module. 2000mAh(could be even smaller), the most commonly seen one, is good enough to provide the correct voltage range even during 2 Amp surge.

I have been using 3.7v Li-ion Battery solution for over two to three months with sim800c and it just works perfectly.

That means we have two 18650 batteries: one for the Esp8266 and MPU-6050, one for the sim800c. Here is the final schematics:

Case Design

Here is the final wires of the device:

As you could see, the wires are pretty messy. When I try to do a mini test, I found it was so hard to put all the wires/modules inside a case. Also, there is a problem with putting the sensor(mpu-6050) on the spring coil of fishing set and connecting to the MCU inside the device case with non-flexible DuPont lines. The mpu-6050 chip is not easy to bind onto the spring coil and get a steady and same position/angle every time. What’s even worse, 4 wire of Dupont lines imposes a constraint on how we arrange the fishing set and bite alarm device. It is not like regular thread like sewing one which is soft and flexible and easy to twist and extend. Lastly, not only does the device box(the case) need to be waterproof, but also the sensor MPU-6050 needs to be waterproof. In software terms, as what Uncle Bob Martin would say in one of his books Agile Software Development, Principles, Patterns, and Practices, those modules are not loosely coupled and highly cohesive - They expose too much its internal details that caller shouldn’t just be bothered.

If we just revisit why we choose the MPU-6050 as the sensor at the very beginning， given we have chosen the onshore strategy, it looks like it could be replaced with an easier one. The fish tugs the fishing line, how could we use that force to trigger something? If you put “sensor” on the Taobao or Ebay, you could find lots of sensors with different purposes. A YL-99 collision switch senor(2.4RMB - $0.4):

(The yellow rubber is not needed)

A collision switch could detect force on it and when force is strong enough to push it to be closed, it outputs a low voltage (0), otherwise a high voltage(1).

With this collision switch as the sensor, we could pack the whole modules inside a standalone case then use a sewing thread to connect the device to the fishing line. Now the connecting way is easy and flexible, hence we could put the fishing sets and device anywhere separately I’d like to based on the environment.

Next is how to deal with the messy rampant wires to make it neat and tidy so that it could fit inside a case as small as possible. First, I bought different standard sizes of the breadboard and try how to fit different modules inside it. Second, I measured the approximate dimension(11cm X 15cm X 3cm) that it would take and bought different sizes of PVC plastic cases. And none of the cases would perfectly fit, so I have to cut it with scissors and drill a hole with soldering iron in the side of the box to put the trigger line through.

The new schematics and breadboard wiring sketch:

Here is the finished case:

Demo Test

Then we need some tests to see whether or not it would work and how long the battery life could last. Here is the demo video I record to test:

Imagine the left side is the fishing line, you just tie the device to the fishing line with sewing thread. (The yellow rubber is not needed)

There is a saying in software engineering that when you try to demo to other people, it usually breaks. So I brought it to the office and did a presentation to my colleagues:

I made some adjustments in the init.lua. First, when it starts, it sends a txt saying init to the backend, then sends every 10 seconds for the first 5 minutes, so I could know whether it is properly set up and running ok. Then it sends a heartbeat request with a collision detection result every 30 seconds. The collision poll is every 20 milliseconds, if once the collision is detected, it will call me in an interval of 0, 2, 4 minutes - three times.

-- init.lua
dofile("sim800c_wrapped.lua")
dofile("collison_wrapped.lua")
pin=4 --D4 some visual clue
gpio.mode(pin,gpio.OUTPUT)
gpio.write(pin, gpio.LOW)
tmr.delay(3 * 1000 * 1000) --delay 3 seconds
sim_setup()
gpio.write(pin, gpio.HIGH)
data = collison_read_if_collided()
sim_send("init colided:"..tostring(data))
tmr.delay(1000 * 1000) -- 1 seconds
print("all done, kick off loop");
gpio.write(pin, gpio.LOW)
counter = 0, called = false, shouldCall = false, callCount = 0

sim_timer = tmr.create(), poll_timer = tmr.create()
sim_timer:register(10 *1000, tmr.ALARM_AUTO, function()            
    if called == true and shouldCall == false then
        print("should call phone now, after 2.5 min and 5 min");        
        if callCount == 0 or callCount == 5 or callCount == 15 then 
            sim_call()
        else
            sim_hangoff()            
            sim_send(counter..',collided='..tostring(true))    
        end
        if callCount == 15 then
            shouldCall = true            
        end 
        callCount = callCount+1 
        gpio.write(pin, gpio.LOW)
    else
        sim_send(counter..',collided='..tostring(data)) 
        gpio.write(pin, gpio.HIGH)   
    end 
    gpio.write(pin, gpio.HIGH)
    counter = counter+1
    if counter == 30 then        
        sim_timer:interval(30 * 1000); 
    end
end)
sim_timer:start()

-- 1000 millseconds 
poll_timer:register(1000, tmr.ALARM_AUTO, function()    
    data = collison_read_if_collided()    
    if data and called == false then
        called = true  
        callCount = 0                   
    end 
end)
poll_timer:start()

I also installed the JuiceSSH(a free SSH client for Android) on my phone so that I could check its long anytime anywhere. And the log would also give me a clue if the call is not made somehow.

But with 20 milliseconds poll interval(I assume the force is applied in a very short amount of time) and heartbeat every 30 seconds, the 2500 mAh battery for powering up the MCU gets quickly dyed out, however not for the Sim800C. The polling frequency is just too high. But if we slow down the frequency and do a quick tug, the collision sensor won’t even detect it. How could we do with that?

There is a similar topic in algorithms: Time-Space Trade-Off. To solve it, either in less time and by using more space, or in very little space by spending a long amount of time.

Yes, we gonna use the extension spring, to use its elasticity to create some tension to absorb the strength or force to buy us some time for detecting intervals. Here I add one rubber band on the end of the sensor length line. With this, I could adjust the polling interval to 1 second (1 second is too high, this could be 2 seconds even 5 seconds or 10 seconds in my mini-experiment but this number should be set by the real scenario later but yeah this tweak does show it is possible to detect correctly in a long poll interval).

223.104.210.120 - - [11/Dec/2021:14:52:07 +0800] “GET /api/dashboard?time=3878805&idx=init%20colided:false HTTP/1.1” 301 169 “-“ “SIMCOM_MODULE”

223.104.210.120 - - [12/Dec/2021:16:36:14 +0800] “GET /api/dashboard?time=305340902&idx=542,collided=false HTTP/1.1” 301 169 “-“ “SIMCOM_MODULE”

Polling every 1 seconds from switch; sending every 3 minutes heartbeat - 2500mAh - it could last like 25-26 hours; for 3200 mAh it could last 36-38 hours. Not too bad!

Real Test

With the all work done, it is a good time to put it to the real test. I happened to have two-three days off and so I went back home. And in that reservoir, the water has dropped down so much, so the rice field side is just full of mud and I just picked the mountainside for fishing.

I cast out, connected the fishing mainline to the staked tent pegs, then link the mainline with the device using a thread, and carefully disguised the device and tent pegs with rocks on the top of it. I looked at the ssh console, all good, so I just went back home around 3PM afternoon.

I’m pretty happy with the stealthiness. You basically couldn’t recognize it even if you pass it by.

The morning of the second day, around 7:50Am, I got three calls on my phone.I was pretty excited and rushed to the bank imaging how big the fish could be. By the time I got to the bank, I was just jaw-dropping to see three or four ducks sitting right on the spot where I set up my device and trigger. I was like “oh shit, no way” :)

I went to check my lines and bite alarm device. It turned out the fishing line and the trigger thread got twisted in total chaos.

No wonder it got trigged! I did not notice those ducks that afternoon when I set up. Then I changed with another pair of batteries and moved the whole set to another venue that ducks couldn’t reach easily.

That later two days it turned to be blank. I wasn’t disappointed since I knew this was a brand new venue, had only one set of rod&reel&line&rig, and esp in winter, it was pretty tricky to get a bite. Instead, the process of trial-and-error has so much fun and I definitely will keep trying it out if later I have another holiday. Keep tuned!

Improvements

There are some possible improvements we could do. Add extra 20 meters lines between the tent pegs and mainline to give it extra buffer when the fish is tugging the line. Also could add some rubber band to the tent peg to counter the force of the fish to simulate what the rod does. The lead and sinker need to be slidable within some range if not droppable, to alleviate the panic of the fish after getting a bite.

The polling style could be changed to using interrupts. Together with the sleep mode of Esp8266, you could make the battery lasts a very long time. But the downside is you won’t get the heartbeat http request.

Possible replace 2G with NB-IoT. No need a real sim card, you could just use e-simcard. So you won’t be too worried when your device is stolen by someone else, therefore, they got your sim card and inserted it into their phone and do some bad things. The downside is it can’t make a voice call (could be resolved by calling Twilio api to make calls from them) and the module SIM7020C is a little bit more expensive.

Prices

Only the battery part is kinda expensive (over 20RMB). Actually 2000 mAh for SIM800C and 2600 mAh for MCU is good enough.

Source Code and Datasheets

Source Code on Github (C&Lua)： tuo/auto_carp_fishing
ESP8266 Technical Reference
SIM800C Datasheet
MPU6050 Datasheet

References

Some good blogs and forums:

Some good youtube channels:

Some good articles:

Auto Carp Fishing Part 1 - Catch Big Carp In China

2021-12-04T04:55:32+00:00

I always have the following questions and impressions on fishing big-size carps:

winter - cold freezing? (bivvy is not even good enough so people just hang up their rods)
not that much time for day session? (need work and away from the big city)
the second half of the night (00:00 - 07:00) - Low temperature, low metabolism. what’s the behavior of carp? any bite? how likely? It is said that larger fishes are more active at night and dawn.
wild water? big lake nature water (not small pond or commercial fishery or syndicate)
simple baits will work? need for fancy and expensive commercial baits?

I only have some pole fishing experience, but not that much. In China, pole fishing is the prevailing fish style. People like the feeling of an adrenaline rush when there is a fish on your pole. I still remember how wicked and satisfying the feeling is when I got a grass carp like 7-8 lb using 5.4 meters long pole and sweet corn.

Left side is the grass carp(3-5lb/7-8lb), top right is the crucian carp (2lb, weight over 1lb is considered as big one for crucian carp), bottom right is the pond)

But when I switched from pond to nature water like a big lake - reservoir, I had so many blanks. I start to realize how difficult it is to catch a big size fish there. And the pattern in pond fishing is quite different from wild water fishing. No one is feeding those fishes and density is very low. So most people go to high-stocked recreational fisheries or syndicates for fun instead of natural water. The low density is because fishery resource here is kinda getting destroyed somehow. There used to have all kinds of destructive fishing practices to catch the fish - e.g - electro、net of mesh with small holes 、firecrackers - which is not sustainable. But it is getting better as life is improving and people and the governments start to realize they need to protect the natural resources(An unprecedented 10-year fishing moratorium was issued in January 2020, covering 332 conservation areas in the Yangtze River basin.) It is a process.

Good nature view and there also has paragliding there. Top left I’m trying fishing the bighead carp with a 5.4 pole. BTW: I’m coming from the central part of China, Hunan Province.

And I could say that fish that with a weight over 1.5Kg ~ 2.5KG could be called - BIG size fish here. There are some fish, sneaky ones, with a weight of over 20kg - 50kg (last time, one of my friends caught a grass carp over 12KG~24LB using a pole in this reservoir), but that’s pretty rare. The type of carp here is almost all grass/common with super rarely you could get a mirror, let alone any black/scaly carp type.

After I observed from and talked with those local anglers here, they are angling with 8.1 meters - 10-meter long telescopic pole, prefeed/prebait with feed corns like every day in the same swim, then start fishing all the day. Going blank is not uncommon, and they might catch a big one after waiting for weeks. Sometimes they are also trying surf rod to casting out with long distance with pack bait and rebait every two hours but still no luck. There was a time when I tried fishing with 5.4 meters pole with corn on the bank of the reservoir, I saw two anglers nearby me using 10 - 12 meters pole just keep baiting with a spoon and rebait their hook and the whole afternoon, no single bite and they ended up with just playing their phones :)

Winter fishing under minus 20 degree, no kidding! That’s the angling spirit of Chinese anglers! No bivvy, no coffee maker, no tea kettle, no barrow, just a kinda genuine and pure form of primitivism in the name of Dionysus :) video source: https://v.douyin.com/8JRKLm8/

An idea just popped into my head. How about making this waiting process kinda non-attended? Then in rainy days or on snowing/windy days or cold times like the second half of the night and dawn, the fishing is just happening there and you don’t need to stay on the bank (even though surf rod fishing could be more comfortable and less concentration required comparing to pole fishing). So your time could be freed up to do other things and when fish got a bite, it triggers something and notify you via something like Wechat/Telegram or just a call - like interrupts in MCU(Microcontroller Unit).

Nowadays Fishing Industry/Vibe

I still remember when I was a kid, I use a self-made bamboo pole and earthworm that dug out of my backyard to catch fish which is pretty effective and simple. Nowadays it is jaw-dropping to know that fishing gears and equipment are getting so complicated and segmented, coming with so many variations. Take a look at the pole fish gears in China and correspondings in the West:

Pole fishing gears and its prices. Click to see bigger picture

For pole fishing: A 7.2-meter telescopic pole is at least 300 RMB for moderate quality, 1000RMB0-2000RMB for good ones.They are tons of components and accessories: fishing platform、multifunctional fishing box、fishing bucket、fishing chair、fish umbrella/brolly、fishing bag etc. The fishing platform is especially heavy if you’re buying a cheap one made of metal.

How anglers in China carrying around with their gears. Gonna improvise by turning fishing pole to carrying pole in the top left - a good way to test out its quality though :)

Those gears add up lots of weight and they are not easy to carry around. Look at the mountains and rice field around that reservoir, I’m pretty sure by the time I got to the bank, I’m already exhausted. I thought the Chinese way is somehow maximalistic. Guess what? Western carp fish is just a little bit more maximalistic :)

ps: I heard Meister Eckhart whispering: The more we have the less we own.

Fox Carp Fishing Youtube: CARP FISHING IN WINTER (Catch more on cold days)
A Nash Trax Evo MK2 Barrow could cost you £279.99 = 2300RMB. Battery supported Nash Power Barrow around £900 = 7000RMB! That angler must have the most anglingg spirit given how weak he is and have to use a battery powed barrow to move around :)

A wheelbarrow - I thought angling is about minimalistic and keeping it simple until some Fox/Korda/Nash/Gardner/Mainline angler of carp fish comes with a wheelbarrow and point to all kindas gears(mat、bivvy、camo cool bags、 bait belts, coffee maker、tea press、kettle、stove、dinner set and cookware sets, etc) inside it smirking and saying: “Oh my Gawd, dear comrade，my young bloke, how dare you? I’m the true minimalist - just one item - the barrow! Horray :)”.

Surely, good types of equipment, tackles, and gears could bring better angling experience, some edges, and more comfortable when angling - unless you know what’re doing. Like a double-edged sword, it comes with pros and cons. Those types of equipment make it so cumbersome to move around, even deflated, just take a look at the mountains and rice fields in my case - how steep the slope is in the mountainside and how close those rice fields are laid out next to each other like a grid.

ps: Even though I’m pretty much a minimalist with just a single fishing bag and a washbasin for fish and as a possible trash bin, I still feel a little bit exhausted when climbing up and downs. Don’t tell me that somebody fancy carrying a barrow of heavy gears around in here. Hope someday Nash/Fox would come up with a helicopter for anglers.

Those commercial tackle companies mainly target recreational fisheries like ponds or farm-raised syndicates, not for wild fishing. Isn’t recreational fishing supposed to be simple, lightweight, and relaxing instead of a …a barrow? Early Church Fathers and later Meister Eckhart, must feel kind of offended by the Roman Curia’s long and boring church services, ceremonies, and rites (In nomine Patris, et Filii, et Spiritu Sancti. Amen - people just couldn’t bear with it, so much so that they coined a word with the last two words - Sancti. Amen - in Spanish for it: santiamén).

Unless you’re a professional angler or angler that have a lot of time, just fishing for the fun and blending in with nature to be like a Dionysian of Nietzsche during the precious break, then fishing in wild water is the no-brainer. And it doesn’t need top-notch gears/baits to get started.

Here is my humble opinion on why?

Fishing Explained

“In our reasonings, there are all imaginable degrees of assurance—from the highest certainty to the lowest species of moral evidence. A wise man, therefore, proportions his belief to the evidence.” - David Hume

Fishing is not rocket science(see hard science and soft science). It is pretty much like sociology and economics - dealing with probability and statistics. it is very hard to do a controlled experiment because there are simply too many factors involved and isolation becomes pretty much impossible to do so. That’s a pretty important shift in perspective, just like investment and sports, from 100% certainty to uncertainty and odds of randomness.

According to Freud (1915), the unconscious mind is the primary source of human behavior. Like an iceberg, the most important part of the mind is the part you cannot see - Freud and the Unconscious Mind

Now think about the catch rate and its factors:

Catch Rate = water temperature + barometric pressure +sunny/cloudy + wind(Windward/leeward) + season/timing + water bodies(pond/rivers/lakes vs still/flowing/creek/current) + waterbed(gravel/weedy) + terrain(depth/basin/drop off) + food source + bait flavors/scents + speice of fish + ... 

There are just a whole lot of levels of randomness inside it.

Factors like the iceberg above the water are the ones we could observe from our experiences. Some factors are more general and universal, some are quite different to different species of fish. Those are the ones at least we could study and learn to figure something plausible out like patterns to improve our chances. To some extent, fishing is a bit rocket science and you can do your homework with reasoning. You can get wisdom and theories from reading books、watching videos、taking with experienced anglers - a very effective way to build up your understanding of fishing. A good example would be fish anatomy and behaviors patterns - know the fish you’re angling at - is it bottom fishing or surface fishing? what water column it prefers? how does it feed? how it find food? what temperature does it like? etc. Even for carp, there are so many different kinds of carp and its behavior might vary a lot, e.g, it would be an extremely slim chance to use corn that grass carp likes a lot to get any bighead carp a bite. There are some nice carp underwater fishing compilations on Youtube like <Underfishing Best carp underwater fishing compilation 2020 (High quality)> showing how carp got caught in front of a camera. Also a series of Carp Fishing: Understanding Underwater from Rob Hughes is pretty helpful.

“Read not to contradict and confute; nor to believe and take for granted; nor to find talk and discourse; but to weigh and consider.” - Francis Bacon, The Essays

The iceberg under the water is a big X - sometimes we called it “luck” - it shows the limits of our knowledge. Its randomness is beyond our understanding. It reminds us that our logic/reasoning that is built on top of it is inherently not that stable as we might think and open to changes from underneath it. Hence we could answer some questions like “what one single factor could stand out significantly over those in this equation?” The answer is no. Unless you fish in a controlled fine-tuned condition like a vacuum pond, you might have one factor over others. But that just applied to that specific case, and it would be practically impossible to determine with wild fishing.

No silver bullet for software engineering, no magic chart pattern of technical analysis for investment, no magic bait, no super bait. Additionally, it reveals that those best practices or golden tips or rules of thumb from others are just their summaries of their past experience which are not tested by your personal experience. Their context might be(for sure) different from yours subjectively and objectively. You can not have two same icebergs, can you?

To turn other people’s experience and theory(outward) to yours(inward), you need to do it by yourself, in your experience. How? through trial and error and learn from it, remember no books or videos, and nothing could substitute for your personal experience. Don’t get me wrong, those guidances serve greatly as a starting point or sth to fall back on when you just have no clue, but never it should be taken as doctrine or sth forever true. Fishing is a dynamic process and just about being personal. If you want consistency in longer time over randomness, then you cant count on the beginner’s luck or find some magic shortcut. You should do your homework - for example like what Joel Greenblat said in «The Little Book That Beats the Market» - and put in effects and times will build it up. Once we know the probability - nature of randomness, get yourself right in one go by buying expensive stuff from or imitating those - in Max Weber’s term, Charismatic authority alike - masters displayed in the videos of commercial tackle companies simply not gonna work. You want to do trial and error many times in a budget and affordable way to figure out your fishing style and strategy that fits your specific case.

“Truth is not to be found in a book. Independent inquiry is needed in your search for truth, not dependence on anyone else’s view or a mere book. Furthermore, such a book merely presents another barrier to progress in your search for truth.” – Bruce Lee

It sounds like software development/lean startup/investment and other things, the idea is the same: keep agile, fail fast, embracing changes.

:

Do your homework(Fundamentals) + Luck

Dedicated + Open-minded

The dynamic and probabilistic nature of the fishing process means you need to prepare yourself up with a good mindset. Sometimes, you not doing your homework but still great luck, turning out to be a good catch. At some other times, you did your homework hardly but no luck turning out to be fruitless a blank. That blank is very common for fishing big-size fish in natural wild water, like gaussian distribution for studying human behavior in sociology, just think about how many extremely tall guys you have met. To illustrate that, I put a Galton Board to get an idea of the normal distribution emerging from a random drop of ball-bearings.

ps: next time, when you buy lottery and hope to win, ask yourself this: Am I gonna get struck by lightning right now? source： Galton board form PhysicsFun: https://www.instagram.com/p/B5qLb2DlZPA/?hl=en。

Big size carps are more spooky and pretty smart in nature water. Just like anglers need to put in work and wait for all planets to align to get a bite, those fishes also need to have all the X factors aligned perfectly to grow to that size.

The unpredictability is sth you can’t control, but the process is sth you can control kinda and be enjoyed. Angling always has offered something to learn about, through it, you learn not just that - fishing outside - but also yourself, and build up some awareness of your identity (self) and appreciate and reconnect with the beauty and serenity of nature. To me, I like fishing in nature water, and that is relaxing.

So I’d suggest those commercial tackle companies could put a one-liner as a disclaimer to show their frankness/deep respect/sincere care for fishing in their promotion videos on youtube: "It depends!"

Fishing Scenario

some requirements and conditions of this fishing case:

reservoir: low stocked big nature water - wild water
target: big size fish with 1.5KG - 15KG
time: 1-2 days consecutive
unmanned/unattended: automatical
baiting/feeding strategy: pre-baiting/feeding once and no halfway rebate
stealthiness: device should be small enough to be camouflaged so that it won’t easily get spotted from the bank
mobility: device and whole fish tackle should be easy to be assemble or disassemble - easy deployment

Quick Summary: prebait and hook the bait, cast out, take down the fishing line from the reel, knot with tent pegs on the bank. Then set up the bite detecting device, link the trigger between the fishing line and device. Leave it for 1-2 days. When got bitten by fish, the strength of fish fleeing way on the fishing line will trigger an alarm, hence giving a call to my phone so that I could just go there and pick up and land the fish.

But we also need some heartbeat to know that the device is running properly. Every one minute, it sends a HTTP request to my backend server which logs its timestamp and whether it got triggered or not. With this data, later we could know what time it got a bite and what temperature it is.

Keywords: wild water、big-size carp、automatical, winter、night, and dawn(fish behavior).

A quick recap: As previously said, I have seen lots of local anglers(some of my friends) spending lots of time on the bank just waiting for a bite using the pole. And in carp fishing of UK/US this got a little bit better with bivvy/coffee maker/cookware etc esp in the raining and snowing days but just too cumbersome&expensive to carry around. Still staying up the whole night in the wild doesn’t seem to be a relaxing and comfortable experience. Fishing in the second half of the night is pretty rare to see. It is just so intriguing to me and possibly a quite new territory to explore about fish’s nocturnal behavior. That’s exactly why I got this idea: non-attended carp fishing.

Before I jump right to the automatical part - trigger device and how it works, like an old saying “A beard well lathered is half shaved”, I need to analyze what kinda fishing strategy would fit in the overall big picture.

Fishing Strategy

I learned many tips and techniques from European carp fishing through videos and books. However carp fishing in China, esp wild carp in wild water - in my case a big reservoir - its behaviors are quite different from European style as Enourpean are primarily fishing in stocked ponds either commercial fishery or syndicate that farm-raised carp has no predators. The fishing resource in China is worse than in the US. I have watched some US carp fishing videos(some Chinese people cane poling in California and catch lots of carp fish with good size), which its wild water is more similar to my case but the density there is still much higher than here. I dare challenge those carp fishings masters to come to China and test themselves on the real battlefield :)

A not very accurate and kinda close summary of fishing conditions in three regions:

| Region     | Water           | Density         |
| ---------- | --------------- | --------------- |
| UK(Europe) | Paid fishery    | High Stocked    |
| US         | Wild/free water | Medium Stocked  |
| China      | Wild/free water | Low Stocked     |

Ps: US fishing style is the closer Chinese one, not that fancy like UK’s. I prefer to watch carp videos from US anglers as their fishing strategy(more lightweight) and conditions(wild water) and personal journey(no so-called masters just individual anglers who likes fishing) are kinda similar to ours here, instead of the UK where lots of tackle companies(commercials are too on-the-nose).

As said previously, there is no secret bait, no magic gear that will surely give you a bite. Even if you have the best rod and reel、most expensive bait、fanciest and most elegant barrow in the world, all that stuff doesn’t make a lick of difference if you’re not putting your bait in front of the fish.

One big factor of the consistent catch is just finding where fish are, esp in the winter. Location is crucial!

Swim/Location

The first thing you need is to analyze the terrain of where you fish - read water. Either take a walk around the water area, or search its google map satellite image, or ask from the local angler, just try to figure out the general terrain of water you’re gonna angle at. No rush to drop your rods. For picking the right swim/spot, it is always good to have direct eyesight of fish rollover or bubbles from stirring mud.

The rectangle area is a fishing road which Darrell Peck found through google maps. You could see its color is different from two sides. source: Korda Thinking Tackle OD 4 EP2: Danny Fairbrass & Darrell Peck | Carp Fishing 2021

Generally, you want your fish spot is closing to snags or something with structure(weed、fallen/sunken trees、overhanging vegetation or rocks) that provide safer places to hide for a school of fish instead of open water. In summer, the fish might spread all over the water and location might not be that important, but in winter, they tend to school up and it is up to you to find the fish. Carp fact: carp is cold-blooded therefore their movements and activity are dictated by water temperature.

Also, carp are spooky and very sensitive to noise, which is why most anglers here prefer to fish in the evening or night in the summertime as much as they can despite some annoying nocturnal animals like bugs and insects, and possibly snakes. Commonly, you want to stay away from those swims with high angling pressures from the bank.

The south side is the deepwater area with mountains sliding into the water very sharply, that north side is kinda shallow full of debris left from submerged rice field. Most local anglers will pick the south side with deeper open-water and fewer snags but kinda rocky at the margin, and I barely see anyone fishing on the rice field side. But I decided to go from the rice field side where I heard a big sound of fish jumping. Other reasons are:

Even though it has snags like the debris of rotten rice plants, it doesn’t do that much damage compared to a rocky structure, considering it is unmanned and unattended. When the fish got the bite, felt the pain(I’m not sure it is called pain, I guess “danger” is better here?), it is gonna drag the line to wherever it felt safe - which mostly is full of snags or deep water so to say. A rocky structure would be much easier to cut the line due to its sharpness than some softy rotten debris could. I forgot to mention I bought a 100-meter long braided fishing line of line size 8 with 8 strands - maybe 4 strands is even better here - which should provide a pretty good abrasion resistance against aquatic plants.
Helicopter or chod rig could help lift the hook from the lake bed and present nicely without being buried in the debris.

Water Depth

Knowing how deep the water is can be essential to fishing. Often we’ve heard people saying: “The deeper the better”, for deepwater could hide big size fish. Likewise, some say “ You should pick margins - where shallow water meets deeper water - a sharp drop-off or deep channel”. Some say “in winter/autumn, go for deep, in spring&summer, go for shallow”. There are lots of different and even contradictory views regarding depth. Our experience tells us that water depth relates to water temperature and oxygen content. But what’s the relationship between those? There are a lot of water depth on fishing activity research on the internet, for example, Effects of Water Temperature and Depth on the Catch Rate and Catch per Unit Effort for Mirror Carp (Cyprinus carpio) caught by Gillnets, could shed some light. Again, it shows that a relationship is pretty complex and can be subject to changes of other factors. In summer, the surface of the water is warm, the bottom is cold; but in winter, it is the opposite.

Generally, big sizes prefer to stay in some depth for safety but shallow water -like close to the bank-is more likely to hold more food sources. Some underwater diving videos show that some fish like to stay in some depth with some structure to hide. That depth depends on water quality、 the overall water depth in the lake and the seasons. You better talk to local anger to get an idea about overall depth like how deep is the deepest part.

Talk is cheap. Show me the code.

All of that seems reasonable but kinda abstract though, can we put some digital/electronic tracking devices/tags on the carps and track its activity, just like tracking whales via satellite tags(Whale Telemetry Group), to hopefully get some insights on its behavior? Big data is the hype, how about some big data from carps?

source: IFishMan Youtube: Wie verhalten sich Großkarpfen in einem natürlichen Gewässer? Hier der August . I highly recommend checking other videos of this youtube channel even though it is in German. Pay attention to its experimental conditions.

Here are some graphics that the German fishery lab did on a natural water pond/lake. They marked different types of fish with different electronic tags(shown in different colors), track their signals of activity across the whole year, and got some interesting data to analyze for its behavior. For example, in winter, you might think the fish will hide in the deepest part of the lake, but it turns out they all huddle together on the shallow border (Reminiscence of …a stock operator - oops, no -the wintertime of elementary school when I was a kid - we all huddle together and push/squeeze each other at the corner of the classroom in an american football way to just get a little bit warmer. kinda like Passenger Squeezes Onto Tokyo Subway During Rush Hour and this People stuffed onto a train in Tokyo, Japan (train stuffing Tokyo) :)

Another interesting observation, we might think as the temperature and oxygen levels drop to the lowest point during the midnight, to that extent, that fish will simply stop all kinds of activity - Sleeping Ariadne, a statue of sluggish. However, the graphic shows they are still moving around during what we perceived as “the worst time of day”.

That just shows how we by default just like to put carp in our shoes. What you think fish should do based on your personal/human experience and feeling != what fish does in its way. You can’t know what fish is thinking(suppose it has thought in whatever you might name it - Tao/God/Natura naturans), instead you could only guess from their behaviors. That’s always a good reminder esp when you’re watching those commercials touting how good they are.

In my case, the depth of the mountainside is like 5-6 meters deep from about 4-5 meters close to the bank. It could be even deeper farther away. On the other side, the shallow side is like 2-3 meters deep from 4-5 meters - since I only do pole fish and never use a surf rod before. I have no idea about what’s the depth of 20-30 - 50 meters away. Given what I know, it seems that 3-8 meters depth could be a good starting point for winter from the shallow side to try out first.

About the methods to plumb the depths, no advanced Sonar or Depth Finder or Fish Finder needed, just use old-fashioned way in cane pole fishing - fishing line with a sinker and a bobble float - cast out a couple of times and adjust bobble’s position along the line to get a ballpark. It is simple and cheap to do but just a little bit time-consuming.

Distance

Along with the previously mentioned old saying “The deeper the better”, there is another saying “The further the better”. And it does make some sense. If you have done pole fishing before, then you know, that the longer the rod and line, the higher chance you could catch big size fish. Why? because often the closest to the bank, typically the shallower the water would be, especially in the daytime, noisier it would be. But how far it should be? is it the further the better? Well not always, you also need to think in terms of the whole lakebed shape and surrounding conditions from a bird’s-eye view.

Fox Carp Fishing Youtube: CARP FISHING IN WINTER (Catch more on cold days)

In the above picture, the center of the lake is deeper than the one close to being the bank. But it is not the best one to fish in the wintertime as it doesn’t have any covers. Covers like reeds/lotus/trees could provide warmer shelter compared to the open water. Even if you couldn’t find any covers, and it is not necessary to just focus on the distance - you have to think about the depth and other factors.

It is not uncommon for some surf rod anglers just to rush to start fishing - in a hurry to put up the bait, cast out extremely long distances without much thought as soon as they arrived at the lake. Afterward, they spend lots of time waiting on the bank, yet they didn’t wanna just spend a few casts to test out the water depth and conditions beforehand. They just go there, mount the hook bait, and cast out wherever they want, then just wait like “I’m done with work on my side, it is the fish - your turn to bite”. Yes, they still could get a bite from fish and catch it from time to time, but just no consistency could be gained.

Think about 100 meters far away with a depth of 30 centimeters vs 50 meters away with a depth of 3 meters given we know the basic depths of this lake. What are the chances of big size carp showing up in those two cases?

It is important to cast out a couple of rods to test depth around the target fish spot - left, right, front, and back - to get a better idea about the lakebed shape near the spot, is it flat, a slope, or a sharp drop-off or basin? Don’t zoom in that one factor too much - you don’t need to cast out to 150 meters away most of the time.

Cane pole fishing has one advantage over fishing with rods & reels, that is it is pretty easier to get cast out accurately in the pretty much same spot everytime. Any deviation on the bank could be magnified many folds in 20,30,50 even 100 meters away. So you need practice casting techniques to gain some accuracy and there are lots of tutorials on the internet. You don’t need the same spot, some offset is perfectly fine. For me, It didn’t take me long to be able to cast accurately like 30 meters distance with 1-3 meters radius in that vicinity. I found it is extremely useful to record the moves of your casting in the video. You could see your posture, movement and adjust accordingly - The Dunning-Kruger effect - what you think of yourself is probably not what you really are.

Also, it helps you get great detail on the lakebed condition of fish spot - is it gravel、rocky or weedy, or silty? How are you gonna choose the right rig strategy for it? Fish might get scared and flee away by the sound of lead hitting the water surface, but some underwater video experiment shows those fish will slowly get back to search for the food. It is not a big deal in my case, as I have lots of time and patience for it while chilling cozily out at home with the heating on and drinking my Longjing/Dragon Well tea - vibe 😎🤞 :)

I have chosen a distance like 20-30 meters far away from the bank. And 2.7 meters long rod with 5000 reels did a pretty good job considering the thickness of the fishing line(line size 8). Since my focus is on fish’s nocturnal behaviors, and night is quieter than daytime, also based on what I know from the above depth section, hence my assumption is that big fish will be more likely to go to shallow water that is close to the bank where it could have more food to feed.

Baiting/Feeding

Basically, how could we attract the fish into the swim? Well, the fish use mostly following senses to detect and identify whether something is good to eat or not: smell、taste、sight. Fish have a pretty good olfactory system to sense any dissolved substance in the water, and they could recognize the smell of amino acids and others having nutritional value. It has shown that carp have thousands of chemoreceptors in their mouth on their barbels and even on the outside of their mouth and under their chin down to their neck. Chemoreception - They could taste, esp for carp, they suck in a big mouth of silt and other detritus - when they smell or feel via taste buds from their barbules, pectoral and pelvic fins of the eatable like bloodworm/corn, etc - palatal taste organ in the mouth trap the food against the bottom of the mouth and expel non-food through gills(almost 90% got ejected). Also, they have sensitive eyes, could see in lower light levels under the water, and detect changes in light and movement above the water - so camouflage like Rambo, be stealth, anglers! However, their habit of feeding in the sediments on the bottom searching for small buried food items has required them to rely on highly developed senses of smell and taste.

Generally, the feeding habit of carp is most fixed and stable like routine - eat what is plentiful - but sometimes they will break it or at least try to discover new food sources. It is a dynamic process. Mostly stable-better chance, sometimes dynamic - slim chance. Consider that the reservoir which has lots of rice field/reed/mulberry tree around it, then most likely the rice/reed/mulberry will get you better chance to get a bite consistently compared to other food like corn that fish never see it before. On the other hand, if you prefeeding the same spot with lots of new food sources, for example, corn, regularly over a long period of time, the feeding behavior of fish might change accordingly. The carp is not one-track minded, they’re open to adapting their feeding habit if it works.

With that being said, it is always good to spend some time beforehand looking at the surrounding fishing location、talking to the local angler to get a good idea about what kinda feeding/baiting feed is best for. Some anglers boast about the effectiveness of some baits with certain flavors or scents. They tend to promote and show how good they are based on their experience so it must be good for others too, without realizing it could be just a cognitive trap or bais: "Rules for me, not for thee".

How do they define whether it is good or not? I saw lots of anglers smell the bait, but come on, you need also taste(just give a chew and expel it out) the bait too, but no one did that :) If the bait smells good for them, then they will think it could attract more fish than those which smell bad. Even though the way that those molecules transmit through the air is different from those transmitting through water, let alone different receptors and mechanisms between human beings and fish. It reminds me when I study English, I tend to give equal timing and stress for all the words in a sentence. Because that’s just how we speak Chinese, which is a syllable-timed language with equal stress sounds. However English is a stress-timed lanuage and have lots of weak forms(function words) to just make the key points stand out more to be easily understood by listeners.

The way we speak our native lanuage is so deeply ingrained into our brain and we don’t even realize that autonomous system. Most of time it works as we expected, but not always. Sometimes you need think outside the box to break that autonomous pattern. Now ask you this: How do you know for carp, which sense is more important and plays a bigger role? olfactory or gustatory ? It seems like most people think the answer is the olfactory sense based on what I previously saw. Well I did some googling and found some research on this topic and in this paper «Feeding stimulants in an omnivorous species, crucian carp Carassius carassius» , it mentioned some experiments on feeding stimulants and it shows that conventional thinking is possibly wrong (at least under that experiment):

Take it as some reference. Do your experiment and try it out, just don’t fall into this automatic and non-examined thinking pattern

Another question is: sola dosis facit venenum - how much is the appropriate one to be defined as a natural flavor and scent level for the fish? It should have some range, isn’t it? Is it always “the more the better”? To how much it could just backfire and get a reverse effect? It is important to know what kind of molecules have either effect. Let alone some flavors or scents or chemicals which work well and stimulate for fish in a separate form, but when got mixed could be a deterrent that has a negative effect on the fish.

Some Wu Wei/Taoism wisdom and a saying from Jesse Livemore - Money is made by sitting, not trading, if you don’t know the mechanism and its chemical of all those additives, then don’t add it. You don’t need to react to that urge to take any excrescent action do sth extra to give your more pseudo aka “Out of Thin Air” confidence or just make you feel better. Maybe It is a good chance ot cultivate some awareness to learn more about yourself.

Feel the same urge of just imitating or copying whatever those glowing/shining - radiating charismatic - masters do in the videos, no need to look up to them and blindly accept everything from what they say. Be skeptical. Put everything on the table and examine it, debunk some myth from those chrismastic authority in a logical way. For bait, you could use homemade ones with lots of options like panko、bread crumb、instant oat meal、grits、cereal、bird seeds、feed corns、grains mixed with some jello gelatin/molasses/beer. Those are natural ingredients and cheap and easy to find. Sweetcorn is very juicy and perfect for fishing during summer. Sweetcorn is my favorite and you could buy the canned sweetcorn and cream for the winter. Boilies are also pretty good which are made of doughs - easy to keep, no stinks, convenient - it keeps small fish away and helps you catch the big size fish. You could also buy commercial ones and it is no big deal. I am not a contemporary cynicism and hate every commercial though :)

Back to my case, rice - from observation of rice field, and feed corn (or maize) - from local anglers, is picked as candidates for the carp bait. Clearly, I don’t want to wait for hours to get small nuisance fish bites, so the rice or pellets or small particles will be excluded. Since I’m gonna leave it under the water for 1-2 days(no rebait), those baits shouldn’t be easily dissolved or lose their liveness like live bait. So no pack bait/method feeder is gonna be used here. BTW, We don’t pack bait with the English way of method feeder. Here we use spring feeder with pack bait much like this US way Outside with Tome: My Carp Rig Setup Spring Feeder for Carp Fishing American Style .

The big solid particles like feed corn(its hard kernel) is a perfect fit and the most important part is it is on a budget and so simple to prepare. Buy some feed corn, soak it and boil it, if you’d like - not necessary- could try an experiment by adding some molasses or alcohols like Baileys or Tsingtao beer or hemp oil later.(Who knows! I have some Turmeric&Curry&Coriander&Garam masala powder left from last time trying to cook Indian food, and I might add those to test out). One downside of it is that adding those sugar or powder additives could make corn ferment quickly and rot faster.

There are a lot of Cinnamomum camphora trees on the mountain. Maybe we could try prefeeding/prebaiting with its ripe seed. It has a very strong natural scent if you leave it for a few days. It is kinda tricky to put it up on the hook as it has a pretty hard kernel

After picking the bait for feeding, the next thing is about the feeding strategy. The most reasonable one is to take a little bit of time to just chum some baits to the same spot every day(like one week or one month) before you get your rods out and start fishing. But it is impossible for me as I only got 4-7 days holiday break and I don’t want to spend too much time on the bank to fish - I only got one hour break each day to go outdoors for fishing(family first) - that’s exactly why I come up with an idea about auto carp fishing. To make it up, I decide to choose one fishing spot and bait there instead of moving around the lake on the bank. Here is my two cents on how to effectively attract fish in big nature water by feeding/baiting:

Basically, we have the fish spot in the center, and feeding, i.e - corn- is evenly scattered around it forming a circle with 3-5 meter radius. Loose chumming could possibly attract further fish in a longer distance than precise baiting, as it could cover a bigger and wider water area by its flavors and scents. Then we use pva bags or pva mesh to precisely bait right in the center of this circle. Well, this seems pretty ideal, the tricky part is how to get chumming like what hands can do but in a long distance? Slingshot、Spod、 pva bags/mesh、 gaint throwing sticks/baiting spoon even fishing bait boats couldn’t do that easily.

Just pay attention to scattering when it is about to hit the water.(ps: who said there is absolutely no good quality one for cheap Chinese product? My rod and reel is like 40 RMB - $7 in total) and I have tried casting lots of times with 100g lead and a pva bag(7x14) of corns yet it still didn’t break)

Self-made Spod: a Gatorade-like bottle、a lead and some glues.

In the above picture and videos, that is a homemade spod with a bottle and a lead. You could see that the corns inside the bottle will be detached from the lead/sinker beneath it as it is falling and is about to hit the water. Those dry corns have a lighter weight compared to the lead and will split out and be scattered into the water - inside a circle of radius like 3-5 meters. The downside is the baiting inside it needs to be light/dry and also it won’t cast out very far away given its bad aerodynamic shape. But it fits my case - actually that’s exactly my intention to not fish from a very long distance - a good compromise.

Rigs

A quick recap on the fish tackles: 2.7 meters long rod($4~27RMB)、5000 reels ($3-$4~20RMB”)、braided line with the size of 8 and 8 strands($2~12RMB)、a slim bobber float($1~6RMB)、80g~2.8oz sinker($0.5~3RMB)、 hair rig of big size 4 circle hook with pop-ups buoyant bait、smaller size 7 Izu Iseni hook for bottom bait with feed corn. To be honest, I have some doubt about using fake bait like plastic corn as bait even though it has a great advantage here: Fish couldn’t steal it, and no need to rebait the hook.

The only southwest part of China in Sichuan and Chongqing use similar fake bait, but not my hometown or province - like never. Most people here rarely use any kinda pop-ups bait or circle hook, instead, they are inclined to use bottom bait with iseni hook for big-size fish. For people fishing with a pole, they only use a maximum of two hooks for each. However, for people fishing with rod and reel, they just use a few hooks. Let’s take a look at two or three most frequently used multi-hook patterns:

I guess in us/uk it is called something like gang/party hooks? (Christmas party in Downing Street while London was in strict lockdown) and The story of 10 Downing Street Christmas Party last December - BBC News :)

The logic is pretty straightforward: more hooks more bites. Despite the so-claimed moral superiority of using fewer hooks from some people - as long as it doesn’t violate the local regulations, I don’t see any problem with that. On the flip side of more hooks is that it means hooks have higher chances of getting trapped in snags or structure forms like rocks/reefs/weed when anglers cast out or play the fish. If you use it in open water which has fewer snags and have a pretty good understanding of the condition of the lakebed, then go for it. But I have a mountain reservoir with lots of snags like rocks and weed, plus a design of non-attended angling, so multi-hooks seem not a good option. Ideally one hook but as I said, I never use pop-up bait with fake corn before, so I want to do an experiment and compare it with common bottom bait with feed corn side by side. Even though technically fish is not eating the fake plastic corn but more like chewing - they just try out the possible food through their mouth and expel it out, I’m still very curious to just, by myself, to catch some fishes to test out popular saying: “If I don’t eat plastics, how come fish will eat it ?”

The hook snap is also linked to the swivel just like what bottom hook does but hook length of bottom one is longer than the hook snap. So when we cast out, the hook snap is gonna take all the force and pva bag won’t fall off.

I’m using pva bags to embed the hook with bottom bait. Since I have two hooks and with different line lengths, to not tangle with each other, I intentionally did not put the inline lead into pvg bag - it also gives more room for bait. Then how to prevent the pva bag fall off the line when casting out (the bag is heavy with corns)? I use a hook snap to hook the top of the bag(the tie part) to the swivel. Just tie off the top of pva bag with extra licks and twist it is strong enough to hold the strength of gravity from hook snap when casting because, by this design, the force is not applied to the hook length line but on the hook snap. I test it a couple of times and it works pretty well. I have a bottom hook presented nicely blending with a pile of corn bait, and a pop-up hook that is a few centimeters away from it. After watching some underwater fishing feeding videos, it looks like big carp always start with outside from the perimeter cautiously. I am intrigued to know which hook it is gonna pick up first in my case.

Now you wonder what’s white powder and oil inside the pva in the above picture? That’s pretty interesting. Some bait companies are touting how magic that bait liquid or like say “smart liquid” that could super charge your bait or whatever. Yet how they could prove it? Here is the video that they show it in the glass water vessel and brag how good it presents: <SEEING IS BELIEVING! Epic Smart Liquid Reaction! Mainline Baits Carp Fishing TV>. Yes, it has some liquid cloud releasing in the water and going up and downs through the different water columns. Looks nice and wicked! But as this comment and author’s reply shows:

Damien Chambers (1 year ago) : 
I’m all up for adding an additional edge to my baiting approach , my main concern in some cases , could this process of a smokey looking halo and rising up to the surface not also have the opposite affect of spooking very wary fish.

Mainline Baits Carp Fishing TV : 
HI Damien - good comment. Throughout the testing period we didn't encounter any situations or problems such as this. General opinion is that carp are likely to be attracted to clouds and this kind of attraction as they are when other fish stir up the lake bed or water inflows and strong winds/undertows colour up water. It'll be interesting to know what you think after trying this out.

It looks like the seller is not very sure and his confidence comes from this black-box “testing period” and “General opinion”. SEEING IS BELIEVING! really? What you see as a good one then it must be good too for the carp? I’m kinda skeptical about that unfounded non-hard evidence. No details or context or data of their testing period has been revealed to back up his views. They just show how great it is visually in water and to me, this is not very persuasive. How big a role it would play among other factors? Does it work in wild water? flow water? What time is the best to add and how much it should add? Yet somehow, indescribably, they - the alchemist - are putting the exaggerated rhetoric in those titles with so much certainty and confidence for other videos of this same kinda liquid.

AVOIDING THE HYPE, UK PEOPLE. DON”T FORGET WHAT FRANCIS BACON&DAVID HUME TEACHED

I did a experiment too. With some research, I have chosen corn powder(500g -6RMB - $1), wheat bran(500g - 6RMB - $1), and hemp oil(250ML - 9RMB ~$1.5) and mixed it and give it the right moisture. Then put it in a pva bag and sink it down in a water vessel and test how it looks and whether it could have those small particles moving around the water columns.

Click the left to watch the video. Look at the movements of the wheat bran. That is like an experiment I did the first time. Pretty sure it could get better later

It looks pretty good and it lasts a pretty long time in that still water tank. Well, I’m not sure about whether it is gonna work or not, but it shows it could get some effects similar to the commercial one. It doesn’t have that kinda halo but you never know whether that halo could make spooky carp shy away. Anyway, the key here is you need to try it out by yourself and tweak it a little here and there to get it to work in your case. And those Smart Liquids are not cheap to play around like what I made. One bottle of this indescribable 250ml is gonna cost you around $14.95 or £12.50 like 100RMB and it is just a liquid additive. Nothing else, mate! How affordable it is!

A poem from Edgar Allan Poe for anyone searching for the indescribable magic bait (Eldorado):

And, as his strength   
Failed him at length,
He met a pilgrim shadow—   
‘Shadow,’ said he,   
‘Where can it be—
This land of Eldorado?’

‘Over the Mountains
Of the Moon,
Down the Valley of the Shadow,   
Ride, boldly ride,’
The shade replied,—
‘If you seek for Eldorado!’

Apart from adding more hooks to increase the odds, adding more rods could also get a greater chance - 3 rods look good (no matter how great you set up your rig and how familiar you’re with the lakebed conditions, it could still happen that your lines and hooks got tangled when casting or got a bad presentation somehow when landing on the lakebed).

Wrap-up

In this article, we have covered the angling strategy that we’re gonna use for the carp fishing, tailed to the unique fishing scenario and non-attended fishing style. Those are the building blocks for the next chapter, which we’re gonna elaborate on the device - how to set up the trigger, read whether got a bite from fish, and call my cellphone to notify.

“The truth and the Way exhibit in the simple everyday movements. Because of this, many miss it (if there is any secret, it is missed by seeking). If there is any secret, one must have lost it by striving for it. The truth is here but men want to decorate the simple truth – the snake with feet.”

“Be Water, My Friend” - keep learning!

My Gears’ Price

“Wait, I’m just from Fox International Products, Korda(I feel Korda offers some cheap ones, not sure) and Gardener. Don’t fool me, show me the real price of those gears. How could it be possible to have such decent quality with low price?”

Well, there you go, in total like 120RMB - $20:

I bought from Pinduoduo(PDD) and Taobao(TB). Actually, the most expensive one is the rigs. I don’t think I need 6 hooks therefore it could be even cheaper.

References

Some good reads:

Some good ones and recommended:

A small pdf from Texas gov: Take Me Fishing: A Basic Guide for the Beginning Angler
Best underwater camera footages of carp feeding and caught!: Youtube Channel: Underfishing
Telemetry of all kindas fishes with good visual information: Youtube Channel: IFishMan
Compared to commercial tackle makers, I enjoy a lot more from an individual’s full angling journey. Catfish and Carp from the US is just such a good yet down-to-earth one. Youtube Channel: Catfish and Carp
Outside with Tom is another angler from US. Fun to watch.
Amateur Angling
Those UK fishing commercials are eye-opening and mind-blowing. Well, even though I am not a big fan of the styles of the UK’s branded videos. Carl and Alex & Fishing Tutorials is practical and fun to watch for beginners. I like the old school fishing, just plain knowdege, a little bit curiosity and common sense. You dont need too much expensive and fancy tackle to catch fish! Graeme’s TA Fishing is a must watch!

Others articles and books on carp fishing:

钓鱼的感想(三): 自动钓大鱼

2021-10-03T04:55:32+00:00

经过第一二部分，已经了解和摸清楚了钓大鱼的一些经验技巧和海竿无人值守需要的准备工作，这部分主要讲解核心部分，展开讲讲如何设计自动检测装置。

这里有两种办法，简单的说：

第一种是借鉴无杆钓鱼的经验。无竿钓鱼就没有浮漂，直接将主线的一端绑在岸边的树根或者木棍地插上。可以在主线和地插连接的地方，加一个弹簧，这个弹簧是倾斜接近垂直地放置，当鱼咬钩拉动鱼线，这个拉力会让弹簧往上抬起来，从而形成一个角度差和瞬间的加速度。假设有这么一个装置可以安在弹簧上，并检测角度差值，当这个角度差超过一定的阈值，就会形成触发，可以认为是中鱼。

第二种是借鉴手竿钓底的经验。手竿钓底，一般都会有一个找底调漂的环节。之后当鱼咬钩，就会拉动浮漂上下运动。一般来说是大鱼的话，浮漂的表现的话就是一个大黑漂 - 浮漂有一个猛烈和迅速的向下的运动，直接浸没到水中。假设有这么一个装置可以安装在这样一个足够大的浮漂的内部，并检测z轴的加速度变化，当这个加速度的变化值超过一定的阈值，就会形成触发，可以认为是中鱼。

这两种各自的优势和困难所在：

第一种的优势：

装置在岸上，不用考虑防水
抛竿流程不受影响，正常抛竿即可
不用考虑风浪，也不用考虑暴晒散热
一旦出现挂底，损失比较小，岸上的装置等不受影响

缺点：

隐蔽性不高，目标还是有点大，容易被过路的其他人和钓鱼人发现拿走，不好隐藏
中鱼信号的准确度有所欠缺，要求鱼线需拉直紧绷，能及时在长距离传递信号
受岸边地形限制 - 必须有合适的障碍物来伪装隐蔽

第二种的优势：

浮漂垂直在钩子的上方，鱼线距离比较短，中鱼信号的准确度高
隐蔽性强，因为浮漂在30-50米之外，其他人无法简单地够着，所以不容易被偷窃
不受岸边地形限制 - 地插可以在水里也可以是岸上，目标小

缺点：

浮漂材质要求比较高，需要既能有好的密闭性能防水也能方便打开，没有类似的现成既有的商品，需要DIY和不断试验
风浪 - 需要在大风浪下，水面有波浪的情况下，保持检测的准确度，同时大风浪意味着需要更大的铅坠来保持定位，这样对鱼竿、渔轮都有更高的要求
散热 - 在密闭性要求高的情况下，夏天中午太阳直射和水面的反光，会导致浮漂内部温度升高，是否电子元器件可以正常工作？
抛竿难度增加，毕竟浮漂本身也有很大的重量加上大的铅坠
一旦线断开，极有可能导致浮漂丢失，这样里面的装置都会丢失，损失比较大

这两者来说，第一个相对第二个在实现难度来说还是简单一点，再一个结合我在夏天拍摄的水库岸边的情况（岸边树木草比较多，适合隐蔽），所以选择第一种方案作为切入点。装置势必要体积要小，同时续航起码需要支持一两天，像上一篇博客简要提到的，包括三个部分：陀螺仪(方向)或加速度(角度)的检测的传感器(输入）、中央微处理控制单元模块（大脑）、无线通信模块（输出）。

当鱼线被拉动，此时弹簧从垂直状态被拉起形成一个夹角，弹簧还可以提供缓冲，防止硬怼硬

看一下模拟中鱼的动图：

ESP8266 微控制单元

MCU，微控制单元也叫单片微型计算机，或者简单点说单片机，是装置的核心模块。单片机，简单理解为微型电脑，特点是体积小，功耗低，有不同的通信模式协议来接入不同的外围设备，比如打印机、显示器等等，互相交换数据。

单片微型计算机简称单片机，简单来说就是集CPU（运算、控制）、RAM（数据存储-内存）、ROM（程序存储）、输入输出设备（串口、并口等）和中断系统处于同一芯片的器件，在我们自己的个人电脑中，CPU、RAM、ROM、I/O这些都是单独的芯片，然后这些芯片被安装在一个主板上，这样就构成了我们的PC主板，进而组装成电脑，而单片机只是将这所有的集中在了一个芯片上而已。

单片机有很多选择，树莓派Rasperry Pi、Arduino(Uno)、51单片机、STM32等等。树莓派太贵，最低两百多，针对我们的场景显得有点大材小用；Arduino也是前些年非常流行的开发板，C/C++语言开发；51单片机、STM32也是出现在各个教材的经典基本通用单片机/开发板，这些也都不错。但是近些年却非常流行一款国产乐鑫的ESP8266系列模组，面向物联网的高性价比（10多块钱）、高度集成的自带WIFI的开发板，比较适合我们这个场景。

这里买的是ESP8266串口WIFI模块 NodeMCU Lua V3物联网开发板 CH340 CP2102中的CH340版本，CH340代表是USB串口驱动的型号。这里后续的代码和操作也都是是在Mac的系统上，先去这里CH340 Drivers for Windows, Mac and Linux下载并安装好对应系统的驱动。用microusb连接板子和电脑之后，去到终端输入ls /dev/tty.*记住对应的设备名称。

NodeMCU 是一款开源的物联网开发平台，支持各种固件系统，比如淘宝上提到NodeMCU Lua V3，也就是基于脚本语言Lua的NodeMCU，还有其他比较流行的MicroPython(Python)、Mongoose OS(NodeJS)等等。这些在系统底层的基础上加了一个解释器，可以直接修改代码而快速的调试，而不用每次改动都编译为底层汇编字节码并上传Flash，后者这样开发调试的速度实在太慢了。

左边是解释器的原理，右边是传统编译上传的终端输出，速度感人

这里要注意到这些解释器并不是完整实现了对应语言的全部特性，只是该语言的一部分。NodeMCU的Lua基于eLua，对应的Lua版本是5.1，不是5.3，根据官方的文档NodeMCU Lua Developer FAQ，math库就没有完整的实现，比如没有atan 反正切函数，导致后续在使用MPU6050计算角度时候就受到这个的限制。同时社区不是很成熟，资料并不是非常完备。

绕过解释器，直接原生的做法也很简单，配置一下Arduino IDE的ESP8266的库和框架环境即可。原生的方式好处在于：不用刷解释器的系统固件；不受解释器的限制，直接C/C++编程，可以利用Arduio非常强大成熟的社区资料，这一点在MPU6050这块角度加速度的计算上特别有用。缺点在于：每次修改代码哪怕只是一个字母一行代码，都需要编译上传，这个速度比较慢，哪怕我已经将Tools -> Upload Speed的上传速度值设置为最大，能快了那么一点点，但是还是费时，整个开发体验还是有点点差，基本是改->等、改->等、改->等的节奏。

但是了，作为新手，还是选择基于lua的nodemcu更加容易上手，开发体验也更好些，后续了解的更多，还是可以方便的切回到Arduino IDE的嘛。

第一步烧录系统。可以去到官方的https://nodemcu-build.com/选择哪些需要支持的模块，选择哪些模块可以去NodeMCU的官方文档NodeMCU Documentation，先了解下基本的原理，了解下基本的Lua的语法，看看自己需要哪些模块。特别是Getting Started aka NodeMCU Quick Start。我这里选择了 bit, file, gpio, i2c, net, node, softuart, tmr, uart，标准的库有bit位操作、tmr时间、i2c是跟MPU6050通信的串口协议、softuart用来跟SIM800C通信的协议。然后去连接ESP8266开发板，打开NodeMCU PyFlasher并选择上一步下载的bin文件，烧录到开发板上即可。

第二步编程调试。推荐的编程环境是ESPlorer IDE，直接运行下载来的JAR包即可。唯一在MAC上面要注意的问题是：串口设备下拉表里无法选择目标ESP8266的设备，需要去设置settings，手动固定死ESP8266对应的串口设备名称。编程的IDE支持相当简陋，推荐VSCode，然后稍微注意下左下角的save/send to esp按钮，用来上传导入或保存lua代码文件到开发板之后，右边的FS info可列出出来当前单片机上的lua文件都有哪些。

要注意的是，nodemcu上电复位后会自动执行init.lua 这个入口文件。如果没有的话，右边串口终端会输出 can’t open init.lua，手动创建一个init.lua,在里面引用其他的模块或者文件-这里是处理mpu6050和sim800C的代码即可，可以参考官方文档的init.lua说明。

NodeMCU还是非常好上手的，特别是自带了wifi模块，可以尝试从简单点亮板载LED到读取外设传感器比如测量温湿度的DHT11温湿度模块（7块钱）的值并通过WIFI上传到服务器上，对照着nodemcu的文档和教程，一步步摸索和熟悉下基础的单片机知识。

MPU6050 - 角度加速度六轴传感器

从上面购买的名字可以看到-MPU-6050模块三轴加速度三轴陀螺仪 6DOF模块 GY-521 六轴姿态，包含了加速度、陀螺仪很温度检测的功能，能帮助做倾斜角度的检测，价格也很便宜5-6块左右，当然还可以选择功能更全带卡尔曼滤波算法的JY61模块，但是目前没有啥必要。

MPU6050传感器是一个集成了6轴运动跟踪装置的模块，分别是3轴陀螺仪和3轴加速度计，同时集成了数字运动处理器和温度传感器。通过I2C总线，他还可以接受来自其他传感器的输入，如3轴磁力计或压力传感器，因此如果将MPU6050与外部的3轴磁力计连接起来，它就可以提供完整的9轴输出了。

要了解MPU6050之前，最好了解下各种串口的通信协议，在这里将会使用到的是I2C和UART。接下里需要了解MPU6050这个传感器，推荐先简单从的输出角速度、加速度和温度值开始，先照着代码敲一遍，跑起来看到输出，在回过头看代码并对照MPU6050系列技术手册datasheet加深理解。

基本上每一个单片机、元器件、传感器都会有对应的技术手册，这个技术手册非常重要，否则心急吃不了热豆腐，容易陷入坑里。

关于I2C可以这么比喻，与之通信的传感器或者元器件相当于一个快递柜（比如蜂巢或者京东），每个快递柜有很多格子，每个格子可以放东西，这个东西可以是快递人放的，收货人去取，也可以是寄货人放的，快递员去取。当你需要给这个元器件设置它的一些属性时，你就往预定的对应的格子存东西 - 写入(Write)。元器件往某些特定格子读取这些设置随后运行中产生了数据就放到了其他的一些格子中。当你需要从元器件读取某些数据时，你就去对应的格子取里面的货物即可 - 读取(Read)。每个格子都有它的编号，这个快递柜也是有它自己的编号。对应过来，这些格子就是寄存器，你只需要去它的数据手册里找到对应的寄存器的说明即可，而快递柜的编号也就是该传感器的I2C从机设备的地址，这个高7位是固定的，也就是默认的0x68，最后一位由引脚AD0决定。

来自MPU6050的技术手册在4.32章节，MPU6050有很多地方需要查看技术手册

找好了MPU6050的设备地址，接下来I2C写入：就是找到对应的寄存器地址往其中写入设置合适的初始值，这里有电源管理、陀螺仪的满量程范围、加速度传感器的满量程范围。后面两个会影响到后续读取出来数值的计算，建议可以对照着数据手册看。I2C读取：主要是陀螺仪数据输出寄存器、加速度传感器数据输出寄存器以及温度传感器数据输出寄存器，分别获取陀螺仪（in degree/seconds unit)、加速度（ in g unit）和温度（ in degree/celcius）的原始数据，要注意它的单位。

-- https://invensense.tdk.com/wp-content/uploads/2015/02/MPU-6000-Register-Map1.pdf
id  = 0 
scl = 7 
sda = 6 
MPU6050SlaveAddress = 0x68
AccelScaleFactor = 4096;  --8g/s
GyroScaleFactor = 65.5; -- ± 500 °/s

MPU6050_REGISTER_SMPLRT_DIV   =  0x19
MPU6050_REGISTER_USER_CTRL    =  0x6A
MPU6050_REGISTER_PWR_MGMT_1   =  0x6B
MPU6050_REGISTER_PWR_MGMT_2   =  0x6C
MPU6050_REGISTER_CONFIG       =  0x1A
MPU6050_REGISTER_GYRO_CONFIG  =  0x1B
MPU6050_REGISTER_ACCEL_CONFIG =  0x1C
MPU6050_REGISTER_FIFO_EN      =  0x23
MPU6050_REGISTER_INT_ENABLE   =  0x38
MPU6050_REGISTER_ACCEL_XOUT_H =  0x3B
MPU6050_REGISTER_SIGNAL_PATH_RESET  = 0x68
 
function I2C_Write(deviceAddress, regAddress, data)
    i2c.start(id)       -- send start condition
    if (i2c.address(id, deviceAddress, i2c.TRANSMITTER))-- set slave address and transmit direction
    then
        print("write")
        i2c.write(id, regAddress)  -- write address to slave
        i2c.write(id, data)  -- write data to slave
        i2c.stop(id)    -- send stop condition
    else
        print("I2C_Write fails")
    end
end

function I2C_Read(deviceAddress, regAddress, SizeOfDataToRead)
    response = 0;
    i2c.start(id)       -- send start condition
    if (i2c.address(id, deviceAddress, i2c.TRANSMITTER))-- set slave address and transmit direction
    then
        i2c.write(id, regAddress)  -- write address to slave
        i2c.stop(id)    -- send stop condition
        i2c.start(id)   -- send start condition
        i2c.address(id, deviceAddress, i2c.RECEIVER)-- set slave address and receive direction
        response = i2c.read(id, SizeOfDataToRead)   -- read defined length response from slave
        i2c.stop(id)    -- send stop condition
        return response
    else
        print("I2C_Read fails")
    end
    return response
end

function unsignTosigned16bit(num)   -- convert unsigned 16-bit no. to signed 16-bit no.
    if num > 32768 then 
        num = num - 65536
    end
    return num
end
function MPU6050_Init() --configure MPU6050
    tmr.delay(150000) -- delay for 150 ms
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_SMPLRT_DIV, 0x07)
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_PWR_MGMT_1, 0x01)
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_PWR_MGMT_2, 0x00)
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_CONFIG, 0x00)
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_GYRO_CONFIG, 0x08)-- set +/-500 degree/second full scale
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_ACCEL_CONFIG, 0x10)-- set +/- 8g full scale  
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_FIFO_EN, 0x00)
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_INT_ENABLE, 0x01)
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_SIGNAL_PATH_RESET, 0x00)
    I2C_Write(MPU6050SlaveAddress, MPU6050_REGISTER_USER_CTRL, 0x00)
end
i2c.setup(id, sda, scl, i2c.SLOW)   -- initialize i2c
MPU6050_Init()
tmr.delay(1000)]

while true do   --read and print accelero, gyro and temperature value    
    data = I2C_Read(MPU6050SlaveAddress, MPU6050_REGISTER_ACCEL_XOUT_H, 14)
    AccelX = unsignTosigned16bit((bit.bor(bit.lshift(string.byte(data, 1), 8), string.byte(data, 2))))
    AccelY = unsignTosigned16bit((bit.bor(bit.lshift(string.byte(data, 3), 8), string.byte(data, 4))))
    AccelZ = unsignTosigned16bit((bit.bor(bit.lshift(string.byte(data, 5), 8), string.byte(data, 6))))
    Temperature = unsignTosigned16bit(bit.bor(bit.lshift(string.byte(data,7), 8), string.byte(data,8)))
    GyroX = unsignTosigned16bit((bit.bor(bit.lshift(string.byte(data, 9), 8), string.byte(data, 10))))
    GyroY = unsignTosigned16bit((bit.bor(bit.lshift(string.byte(data, 11), 8), string.byte(data, 12))))
    GyroZ = unsignTosigned16bit((bit.bor(bit.lshift(string.byte(data, 13), 8), string.byte(data, 14))))
 
    -- ACC in g unit, Temp  in degree/celcius, Gyro in degree/celcius
    AccelX = AccelX/AccelScaleFactor   -- divide each with their sensitivity scale factor
    AccelY = AccelY/AccelScaleFactor
    AccelZ = AccelZ/AccelScaleFactor
    Temperature = Temperature/340.0+36.53-- temperature formula
    GyroX = GyroX/GyroScaleFactor
    GyroY = GyroY/GyroScaleFactor
    GyroZ = GyroZ/GyroScaleFactor
    
    print(string.format("TUO-Ax:%.3g Ay:%.3g Az:%.3g T:%.3g Gx:%.3g Gy:%.3g Gz:%.3g",
                        AccelX, AccelY, AccelZ, Temperature, GyroX, GyroY, GyroZ))
    tmr.delay(100000)   -- 100ms timer delay
end

参考mpuTest.lua

最后看看ESP8266是如何跟MPU6050的连线的：

注意接地和正极VCC别搞错了，否则容易损坏，建议给杜邦线设计专门用途的颜色，比如正极是红色，接地是黑色等等

这里实际中要把这个传感器贴在弹簧上，其实只需要关注acc一个x轴上面即可，每个一秒钟读取数据，观察是否大于某一个阈值。

如果要想实现第二种设置，也就是将单片机放到浮漂球内部，就需要获取在z轴方向的加速度变化，这种情况跟上面的差不多，基本都是简单原始获取的值就足够用了。

如果需要做一些复杂的姿态解算，那就需要复杂点的向量计算。这两个文章《MPU-6050 6dof IMU tutorial for auto-leveling quadcopters with Arduino source code - Part 1》和《MPU-6050 6dof IMU tutorial for auto-leveling quadcopters with Arduino source code - Part 2》非常详细解释了MPU6050工作原理，以及如何过滤噪音，和平衡角度等等比较高级的用法，作者的最终目的用这个来控制无人机，精确度要求高，值得学习下。

可以取其中一段核心代码片段看看：

//source: http://www.brokking.net/imu.html MPU-6050 6dof IMU for auto-leveling multicopters

//Gyro angle calculations: 0.0000611 = 1 / (250Hz / 65.5)
angle_pitch += gyro_x * 0.0000611; //Calculate the traveled pitch angle and add this to the angle_pitch variable
angle_roll += gyro_y * 0.0000611;  //Calculate the traveled roll angle and add this to the angle_roll variable

//0.000001066 = 0.0000611 * (3.142(PI) / 180degr) The Arduino sin function is in radians
angle_pitch += angle_roll * sin(gyro_z * 0.000001066); //If the IMU has yawed transfer the roll angle to the pitch angel
angle_roll -= angle_pitch * sin(gyro_z * 0.000001066); //If the IMU has yawed transfer the pitch angle to the roll angel

//Accelerometer angle calculations
acc_total_vector = sqrt((acc_x * acc_x) + (acc_y * acc_y) + (acc_z * acc_z)); //Calculate the total accelerometer vector
//57.296 = 1 / (3.142 / 180) The Arduino asin function is in radians
angle_pitch_acc = asin((float)acc_y / acc_total_vector) * 57.296; //Calculate the pitch angle
angle_roll_acc = asin((float)acc_x / acc_total_vector) * -57.296; //Calculate the roll angle

//Place the MPU-6050 spirit level and note the values in the following two lines for calibration
angle_pitch_acc -= 0.0; //Accelerometer calibration value for pitch
angle_roll_acc -= 0.0;  //Accelerometer calibration value for roll

if (set_gyro_angles){  //If the IMU is already started
    angle_pitch = angle_pitch * 0.9996 + angle_pitch_acc * 0.0004; //Correct the drift of the gyro pitch angle with the accelerometer pitch angle
    angle_roll = angle_roll * 0.9996 + angle_roll_acc * 0.0004;  //Correct the drift of the gyro roll angle with the accelerometer roll angle
}else{  //At first start
    angle_pitch = angle_pitch_acc; //Set the gyro pitch angle equal to the accelerometer pitch angle
    angle_roll = angle_roll_acc;   //Set the gyro roll angle equal to the accelerometer roll angle
    set_gyro_angles = true;        //Set the IMU started flag
}
//To dampen the pitch and roll angles a complementary filter is used
angle_pitch_output = angle_pitch_output * 0.9 + angle_pitch * 0.1; //Take 90% of the output pitch value and add 10% of the raw pitch value
angle_roll_output = angle_roll_output * 0.9 + angle_roll * 0.1;    //Take 90% of the output roll value and add 10% of the raw roll value

来自上面的MPU-6050 6dof IMU for auto-leveling multicopters

这里有几个函数并不在NodeMCU对应的Lua版本支持之中。How is NodeMCU Lua different to standard Lua? 提到math这个标准库是被忽略没有包含进去的，所以这个sin/cos/asin/acos/atan2都是没有的，只能是自己手动写一个lua的标准math库的函数.

所以这里就体现出来用原始点的C/C++写的好处来了，Arduio社区资料非常丰富，关于计算Roll, Yaw, Pitch这块，论坛里有非常多的讨论和代码，比如Converting Raw data from MPU 6050 to YAW,PITCH AND ROLL、Converting rotation angles from MPU6050 to roll/pitch/yaw、 YAW Calculation from MPU6050等等，都是基于Arduino用C/C++所写；另外有很多代码和功能甚至封装了起来，作为独立的库提供到用户使用，这篇文章里《Gyro (Position) sensors (MPU6050) with Arduino – How to access Pitch, Roll and Yaw angles》编译封装了一个单独的库 rfetick/MPU6050_light，只需要在头部导入#include <MPU6050_light.h>，直接可以通过mpu.getAngleX()、mpu.getAngleY()、mpu.getAngleZ()直接拿到对应的角度，开箱即用。这个在NodeMCU和MicroPython这块，还是没有对应的那么成熟丰富的社区的。

这部分输入检测的代码就差不多实现了，接下来是无线通信的输出部分。

SIM800C - 无线通信模块

如何在野外如何将中鱼的信号发出去了？首先MCU自带的wifi是用不上的，只能是借助手机数据网络。手机网络发展到现在就好几代了，目前是5G，大概的迭代路线是：

G(GPRS)→E(EDGE)→3G(WCDMA)→ H(HSPA)→H+(HSPA+)→4G/LTE(LTE-FDD/TDD)

G,E属于2G，3G,H属于3G，H+,4G/LTE属于4G，速度依次增快。这几种网络使用感受上的最大区别当然就是速度和可以传输数据的类型不同，比如2G主要是语音业务，2.5G就可以传送数据业务，3G达到快速数据业务传输，H使数据速率更高，LTE提高了容量减小了延迟，实现全数据网络等等

对比分析下，就知道我们这个传输数据的场景其实数据量非常之小，可能不到几kb，频率也不会太高，所以听起来2G - GRPS网络就足够了，而且价格还便宜。目前已有的物联网IoT很多通讯网络还是使用的2G，但是现在三大运营商已经开始清退2G网络了给5G让路，所以足够预算够还是建议不用2G，有可能出现买了插上之后没有2G信号。

根据移动最新政策显示，将于2020年底前停止新增2G物联网用户，2021年不再发放2G物联网卡，并将由NB-IoT、4G Cat1/1bis技术承接2G物联网用户。

这个主要是看你用模块干什么了

如果你要是用模块打电话、发短信、上网，你就用4G模块，我知道的有Air720H、SIM7600CE、WH-LTE-7S5 如果你要是没有打电话发短信的需求，只是用来上网，你可以选NB-IoT模块或eMTC模块。NB-IoT模块很便宜，而且国家重点扶持这个技术。NB-IoT型号特别多，你自己上淘宝上一搜一大把。

实际上NB-IoT模块却不便宜，我们对比下基于2G的GRPS模块Sim800c、基于NB-IoT替换sim800c的SIM7020C和4G Cat1的Air724U

结合这个项目需求来看，那还是sim800c比较划算，如果预算够的话，建议上右边两个。

SIM800C可以通过AT指令发送命令，可以先连接电脑，调试一下基本的指令。这个时候需要一个USB-TTL调试模块、一个手机sim卡、可能还需要一个Nano卡槽和Sim800c模块。SIM800C插好手机卡，然后使用USB-TTL连接SIM800C然后插入电脑供电，观察到电源指示灯亮起之后，用跳线帽或者杜邦线将PWX和GND短接，给到一个低电平给PWX来出发启动模块，这个时候模块才会开始工作。调试时特别注意下LED闪灯频率，这个可以区分模块的工作状态。

这里Mac系统的串口调试工具建议使用CoolTermMac，设置好串口设备的端口(Serial Port Options)的Port、终端断行模式(Line Mode)和View ASCII文本显示。

特别要注意SIM卡和卡槽的方向： AT+CPIN? 可以查看SIM卡是否准备就绪，如果返回：ERROR，要么SIM卡插入方向错误，要么SIM卡异常。同时请确认手机卡是否启用了无线网络，否则http请求无法成功。

SIM800C可以接受哪些AT指令，建议大家参考下它的数据手册，但是中文的数据手册在线的没有找到，只有这个勉强能用《SIM800C系列用户使用手册》，一般来说你购买的淘宝店家应该会给到你一份手册。我们这里用到的包括打电话和每个一分钟发送一次MPU6050采集的角速度加速度温度信息到服务器方便后续排查和调试。下面是连线示意和SIM800代码：

MCU - SIM800C

RX(D3) - TX

TX(D2) - RX

GND <-> GND

VCC <-> VBAT

代码：

---- Create new software UART with baudrate of 9600, D2 as Tx pin and D3 as Rx pin
function writeCMD(s, cmd)
    print("\n"..tostring(tmr.now())..": send cmd", cmd);    
    s:write(cmd.."\n");   
end

function sim_setup()
    if not s then 
        print("\n"..tostring(tmr.now())..": initilized su\n");
        s = softuart.setup(9600, 2, 3)
        s:on("data", "\n", function(data)
          local txt = string.gsub(data, "[\r\n]", "")
          print("\n"..tostring(tmr.now())..": receive from uart:", txt)
          local pattern = "^HTTPACTION"
          if txt:find(pattern) ~= nil then
              print("\n"..tostring(tmr.now())..": RECIEVED HTTP ACTION DONE:", txt)
              writeCMD(s, 'AT+HTTPTERM')
    --          writeCMD(s, 'AT+SAPBR=0,1')
          end       
        end)
    else
      print("\n"..tostring(tmr.now())..": existed su\n");  
    end
end
function sim_send(txt)
    writeCMD(s, 'AT+SAPBR=3,1,"Contype","GPRS"')
    writeCMD(s, 'AT+SAPBR=3,1,"APN","CMNET"')
    writeCMD(s, 'AT+SAPBR=1,1')
    writeCMD(s, 'AT+SAPBR=2,1')
    writeCMD(s, 'AT+HTTPINIT')
    writeCMD(s, 'AT+HTTPPARA="CID",1')
    local url = "xxxx.com/api/dashboard?time="..tostring(tmr.now().."&txt="..txt)
    writeCMD(s, 'AT+HTTPPARA="URL","'..url..'"')
    writeCMD(s, 'AT+HTTPACTION=0')
    s:write(0x1a);  
end

function sim_call()   
    writeCMD(s, 'ATD186xxxx5235;')    
end

代码在gist上面: sim800_http.lua

这里将MPU6050的信息拼成一个字符串txt，然后直接通过最简单的GET方法，发送到后端，在后端可以看到如下日志：

可以看到当我把MPU6050贴在弹簧上，模拟被鱼拉动而将弹簧抬起来的时候，在acc_x上面有一个明显的差值，这个差值多调试几次就可以找到合适的出发阈值。

常见问题 - SIM800自动关机、工作不稳定

然而在实际测试之中，经常会碰到一个问题: 无法稳定正常的工作，或者是搜索不到信号，或者板载的LED闪的频率不稳定，甚至有时候干脆自动关机。这个问题在论坛和youtube下面经常看到，甚至不少人因此放弃了Sim800这个方案。

这个问题，跟编程在运用新框架新技术会碰到的某些问题其实很类似，某段代码或者逻辑体现出来比较诡异奇怪，这个时候一般的思路是将问题拆分定位到某个小的范围，第二个就是回归文档，当你有了一定代码实践的积累，有时候容易忽略文档，但是这些时候往往要慢下来，回归到文档上面，可能有时候能事半功倍，否则容易陷入无头苍蝇抓瞎的处境。

SIM800跟ESP8266 MCU通信是通过TTL电平的方式，也就是高电位是1，低电位是0，这样就可以组合以二进制的方式在双方之间传递数据信息。这就意味着这单片机和Sim800必须有相同的板载电压作为高低压，低电位好理解，都是接地。

查阅ESP8266的技术手册datasheet在第19页的5.1电气特性这块可以知道：

虽然这里是ESP8266的技术手册，但是需要指出的是，后续SIM800技术手册也是非常重要的，里面提到了一些常见的问题和解决思路，需要特别注意的点，里面就已经写的清楚明白了

从上面看的出来，SIM800工作电压范围2.5-3.6V，经过自带的LDO板载稳压之后板子的电压是3.3V。按照上面的连线，那么SIM800C的接入的电压也是3.3V，这时两者的电压才会相同。

查阅SIM800的技术手册datasheet在第20页的4.1供电这块可以知道：

4.1

模块VBAT的电压输入范围是3.4V到4.4V，推荐电压为4.0V。模块以最大功率发射时，电流峰值瞬间最高可达到2A，从而导致在VBAT上有较大的电压跌落。

电源要能够提供足够的峰值电流以保证在突发模式时高达 2A 的峰值耗流。

4.1.1

在用户的设计中，请特别注意电源部分的设计，确保即使在模块耗电流达到2A时，VBAT的跌落也不要低于3.4V。如果电压跌落低于3.4V, 模块可能会关机。

在数据传输突发期间，模块的最大电流消耗约为2A，这个时候就需要电源能提供一个2A的浪涌电流，否则板载电压将会下降（功率不够），导致自动关机。很明显，如果SIM800C的电源是由ESP8266提供的话，不管是电压还是电流都不足以支持突发浪涌的情况。官方的技术手册在4.1供电这块提供了两个解决办法：

第一个： 5V降压LDO

使用5v的电源，用额定2A的DC-DC降压转换器比如LM2596(2块钱左右，可输出高达3A的电流)，将输出连接到SIM800C的VBAT。

第二个：使用单独的3.7锂电池（推荐）

技术手册里在章节4.1.1里提到客户可以直接用3.7V的锂离子电池给模块供电。这里可以用18650锂电池，2000mAh左右或者往上的大容量功率，能够确保在2A的峰值电流下，还能保持正确的电压范围。

这里展示完整的各个模块连接起来完整的样子，同时我们将init.lua修改下加上读取mpu6050的加速度角度值，和发送sim800c的代码，（完整代码在github tuo/auto_carp_fishing)就算基本完成了。

可以看到我们使用两个单独的3.7v的18650的电池（一个电池大概8-9块），一个单独给sim800C供电，一个是给MCU ESP8266单片机；要注意电池连接单片机上的引脚应该是VBAT，而不是VCC。 VCC是板载的电压，设计是不能超过3.3V，而18650的输出是3.7V ，所以要接到VBAT/VIN(5v)这个引脚。

电池续航可以使用多久？

MPU6050每隔2秒读一次数据，每隔1分钟发送一次HTTP请求，大概2500mA的电池可以支持两天没有问题，这个足以符合我们当初设想的要求。关于电池放电这块，需要了解电池的放电曲线measured discharge curves for 18650 cells ，也就是随着时间输出电压是下降的, 当下降到一定电压时，就无法给模块提供足够的电量。

要延长续航时间有几个办法：

换更大的电池比如3300mA的
其他更细的技术的电池比如基于Lipo， 18650是基于Lithium-Ion Lithium-Ion (18650 cells) versus Lipo – which is best?
拉长采样时间 - 比如每隔2秒可以是5秒，1分钟可以是5分钟等等
中断 - 与其不断去询问数据并计算没有超过阈值，可以用中断的思路，设置好阈值，只有当传感器的对应数值超过阈值时，才会跟通知单片机
单片机有低功耗模式 - 可以参考技术手册，根据实际调整

实战的问题

选择第一种方式，将装置放在岸上，是因为其相对而言比较简单一点。我开始考察并有次想法的时候是在夏天7-8月份，可以看一下当时的水库岸边的情况，水库水位很高，离山体一侧的树木和草丛比较近，此时我觉得用第一种方式很好办，比较好隐藏装置，而且岸边没办法可以让人走动通过，所以不存在被别人绊住脚或者轻易的发现。

但是当我买好了海竿渔轮鱼线钩子，并调试好了装置，准备去水库边大干一场时候，当时已经是9-10月份，发现水库的水退了巨多，此时已经水离岸边的上面的树木草丛那块有好大一段距离，都是碎石，人可以很方便的走来走去，架设装置的难度太高了，找不到合适的地点。

无奈，只能选择第二种难度更高的方案。第二个方案有一个关键性的问题就是：装置需要安在浮漂里面，这个浮漂必须容易打开取出来，然后还能放进去，并且在水里能密封放水。

关于这个球，浮漂球，是什么样子，我搜索了一下，只有一个RoboSpace Sphero sprk+可编程机器人教育入门遥控机器球比较接近我的预想。

但是可惜直径太小73MM，都不够放下电池盒子，要完整放得下，直径最少的10CM以上。

淘宝没有现成的商品，只能找到比较相似的是这种球 - 112扭扭蛋盒盲盒外壳圆形蛋高清透明扭蛋球奶白色球形模具可打开，扭蛋机用来装展览展示的物品用的，中间有两个罗文，可以拧上和关闭，但是因为它的作用是为了展览，通用的模型，所以这个防水，也就是中间罗文并不严丝合缝，浸没在水里，毫无疑问肯定是要漏水的。

第一个难题是防水。关于中间拧紧和扭开的缝隙处的防水问题，我想到也许可以使用热胶枪解决。当把装置放入这个球并拧上之后，用热胶枪给缝隙处罗文处涂一层胶，放置一会干燥之后在下水，应该可以保持一定的防水效果。

第二个难题是如何将这个球上到主线上。因为这个球没有一个扣，或者凸起的钩子什么，就是一个光滑的表面，如何跟太空豆连接了。这里需要在球的底部，钻一个小孔，然后用其他的塑料制作一个提环类似的模具，再穿过去卡住，再用AB胶粘住，晾干24小时定型，保证这个环的强度和防水性。

第三个难题是抛竿。因为本身球和元器件特别是两个电池的重量，再加上铅坠和水溶袋，那么这个总的重量是比较大的，对一个杆子的硬度和长度都会提高。同时在抛竿时保证浮球内部组件不松动，所以需要在球的内部安装一些锚点，能较好的固定住电子元器件，保证杜邦线这个连接头能不脱落。

第三个难题是否可以抗风浪和暴晒。如果风浪比较大，容易带着浮漂晃动，甚至拉扯铅坠拖着跑。这个时候只能加大铅坠的重量，这样一来有会碰到第三点的难题。同时在夏天中午高温暴晒和下雨天的暴雨，能否保住元器件是否能正常工作和不进水，这都是挑战。

不过这些都可以在后面慢慢实验，在过程中不断修正，一些工具比如胶水、扭蛋球等等都在路上，后面我也会保持这篇文章的及时更新。

更新后续一

用触碰式开关YL-99替代mpu6050六轴感应- 使用橡皮筋增加缓冲，可以延长碰撞的时间，

223.104.210.120 - - [11/Dec/2021:14:52:07 +0800] “GET /api/dashboard?time=3878805&idx=init%20colided:false HTTP/1.1” 301 169 “-“ “SIMCOM_MODULE”

223.104.210.120 - - [12/Dec/2021:16:36:14 +0800] “GET /api/dashboard?time=305340902&idx=542,collided=false HTTP/1.1” 301 169 “-“ “SIMCOM_MODULE”

poling every 1 seconds form switch; se nding every 3 minutes gps signals - 2500mAh - support like 25-26 hours

淘宝购买的部件清单

没有找到所有的，但是差不多都在这了，总体价格最后没多少钱，基本上都是性价比不错的,PDD里买的物件的质量比我想象的好 :)

资料参考

Source Code on Github 源代码(C&Lua)： tuo/auto_carp_fishing
ESP8266 Datasheet技术参考手册›
SIM800C Datasheet技术手册
MPU6050 Datasheet技术手册
网站: Last Minute Engineers
网站: The IOT Projects
论坛: Arduino Forum
论坛: DFRobot Forum

引用

钓鱼的感想(二): 海竿钓大鱼

2021-10-02T04:55:32+00:00

接上文，在大水面，海竿钓大鱼的优势在于一个是钓的距离远，一寸远一寸强，深水的可能性更高，藏大鱼的可能性更高，但是海竿有个问题是打窝不像手竿那么好打，因为距离太远，打窝勺都够不着；关于海竿竿子的隐蔽性问题，可以用地插加弹簧泄力解决；接下来就是如果化被动为主动，怎么实现检测鱼咬钩的信号并发送到手机端。

理想的自动钓鱼的场景：确定目标鱼（草鱼、鲤鱼、青鱼和大鲫鱼），利用海竿抛竿先打窝，然后在设置好中鱼检测装备，挂好饵料将竿子抛竿打到窝子的附近；抛出去之后将鱼线从渔轮上取下来挂在地插上，回去等待手机中鱼的短信或者电话即可，一般是一到两天的时间，中间就不会动它了，没有人去钓点去检查钓组或者其他的补窝续窝。

也就是一次准备搭建好之后，就不会去干预钓鱼的过程，直到中鱼或者下一次重复。所以准备工作比较重要，需要梳理和分析每个步骤的要点和可行性：

* 确认目标鱼

目标鱼会影响我们的钓法和饵料的选择。钓法简单的说有两种：钓浮和钓底。基本上个体比较大的鱼，鲢鳙是喜欢1.5-2.5米深，半水浅水层，主要是虑食性的；翘嘴也是半水浅水，但是个体一般比较小，嫩玉米等比较好钓；草鱼基本上全水层，从半水到深水都有游动，玉米芦苇等等都是比较喜欢的饵食种类；鲤鱼主要是深水，底部，跟推土机吸尘器一样，吸进来水底部的沙子土腐殖质和食物一起吸进来，然后吐出来沙子土等杂质（这难道是鲤鱼为什么说有一种土腥味，很多地区的人都不吃)，为了确认什么是杂质什么是食物，需要反复的吸入吐出，玉米也是比较常见的钓饵；青鱼应该是里面个体最大的一种，螺蛳玉米都是不错的选择，基本上也都是钓底为主；

钓浮是基本上是针对比如半水层鱼，比如翘嘴、鲢鳙；钓底的话基本上是鲤鱼、草鱼、青鱼也包括鲫鱼。简单的说，钓的浅没有钓的深碰到大鱼的可能性高，比如鲢鳙最好的钓的时候也就是温度比较高的时候，不像其他鱼种，可坐钓的季节和时间更多。当然总体来说，夏天温度高比冬天温度低好钓，因为鱼的活力更好，进食的欲望更高，某些低温的情况，可能最后只有鲫鱼才会进食。

鉴于此，钓的是大体型的鱼，所以饵料和窝料就不能是比较稀的饵，比如不能是酒米或者豆腐渣等那种粉末或者接近类似状态的窝料，在水里太容易融化散开，无法保持长时间稳定的状态，吸引来的更多是小鱼，一下子被吃掉了，无法长时间留鱼或者诱惑鱼进窝。能够在水中保持长时间，比如上面提到的一两天，而饵还能有相对来说不错的状态，就只剩下玉米螺丝这个选项了。那应该是嫩玉米还是老玉米了？一来说嫩玉米相对来说钓浮半水比较多（其实我钓底嫩玉米打窝嫩玉米钓也不错，但是也只是一两个小时，需要经常补窝），嫩玉米贵，而且时间窗口也就夏天那一阵子；相对来说老玉米一年四季都有，煮熟之后加上一些刺激性的诱鱼剂，比较酒糟小药甚至菠萝啤都可以等等，发酵或者浸泡闷个几天，就能在水中形成比较强劲的穿透力，在大水面容易吸引鱼进窝子，相对嫩玉米能更长时间发挥诱鱼的作用。

所以这一步我们确认了目标鱼基本是鲤鱼、草鱼和潜在的青鱼鲫鱼，和窝饵料的选择-玉米。

* 刺鱼的成功率

因为钓鱼过程中，无人值守，不会有手竿那种看浮漂有信号就提竿刺鱼这个主动步骤，所以这一块如何被动刺鱼并提高准确性就是一个非常关键的环节。在这个之前，需要最好有水下摄像来观察大鱼吃食咬钩的动作和习惯，特别是鲤鱼草鱼。

上面截取自世图渔乐，完整视频在：第7集因为抢鲜玉米，两条草鱼打起来了

这个油管的频道很多水下鲤鱼吃食咬钩的视频：Best carp underwater fishing compilation 2020 (High quality)

结合下来可以看到大鱼能长成是大鱼是有原因的，吃食非常的谨慎：

习惯性的从窝边开始，不是想象的直奔饵料集中的区域，而是从比较零散的周边开始
不断的吸入吐出，直到反复确认安全才会最后吸入进食 - 涮饵
当吸入吐出时感受到不安全的因素，比如有股力量比如过长的水草或者不够顺滑入口而造成异物感，鱼第一反应是快速吐出加以大幅度大力度的摆头来尝试摆脱异物，这个力量看起来非常大
第一次摆头之后，还会有后续连着有各个方向上的摆头，力度也挺大的，但是看起来没有第一次大

手竿钓鱼或者海竿有人值守这个时候会反应在浮漂上会是浮漂的上下动作，这个时候人只要去提竿主动刺鱼就能比较好的刺鱼，提高命中率。但是如果是全程无人值守，那么就需要结合上述的观察发挥被动刺鱼的最大威力，争取做到：一击即中。

上面的传统的钓组模式，这也是传统海钓串钩的模式，就不是非常适合了，将玉米穿在钩子上，露出钩尖，钩子和饵料是一体的，直上直下，没有太多角度的变化，难以保证鱼口在进食吐出时可以有机会刺到鱼的下嘴唇，刺鱼的成功率就不会太高。从吃口来看，最好钩子和饵料不是一体的或者说可以有些角度的变化，带一点弧线而不是一条直线，这样可以让鱼在吸入吐出的时候，这个弧度角度可以将钩子轻微的挂在或者带在下嘴唇，这样鱼一感觉异物不舒服有危险时，开始第一下最为猛烈地起身摇头想摆脱鱼钩，给到一个瞬时间猛烈的逃窜力。

根据作用力与反作用力，如果可以利用这股鱼本身施加的力量，比如离钩子不远的线上某个点（八字环附近），绑上一个重量比较大的铅坨比如80-150克，那么这个反作用力会作用到钩上，将鱼的下嘴唇打穿刺穿，这一下的力量足以牢牢的挂住鱼，刺鱼的成功率也会提高 - 这个也就是国外的欧鲤钓钓法和国内四川地区的珠珠钓法。

死铅，也就是固定死不能在线上移动位置的固定铅坠的办法，可以保证第一下鱼大力摆头时，牢牢地刺穿鱼的下嘴唇，但是这个时候瞬间子线所承受的力也是非常大的，所以子线要足够的粗，另外钩的强度要非常好，最好是有伊势尼那种钓大鱼的钩子的强度（否则容易钩子被拉直）；另外一个子线的长度也要稍微长点，但是不能是完全伸直没有余量，因为上面我们注意到鱼会不断的吸入吐出，所以鱼线最好不能绷的太直，否则一个微小的鱼线的张力就会让鱼还未完全吸入就会感觉到线的张力的异样就会立马吐出来，这就难以刺鱼。

再看上面的视频，可以看到鱼在第一次大摆头之后，还会有第二波第三波的摆头，但是幅度和力量都没有第一次那么大；虽然说力量没有第一次那么大，但是也会施加很多力在钩子和铅坠之间这段子线上面，而这一段是整个线组里比较薄弱的；如果可以在第二次三次后面的摆头中，可以让铅坠往主线上滑动，这样可以让钩和铅坠之间的线的长度拉长，这样受力就会被分散的更多，不容易出现子线被拉断同时也可以保护子线，延长使用寿命，同时鱼的嘴唇也不容易被拉豁豁（也就是鱼的嘴唇因为受力太大时间太长，导致嘴唇直接被钩拉的裂开了，下嘴巴被打烂了，然后跑鱼），进一步提高中鱼率。总结来说，也就是铅是锁死的（第一下有力的刺鱼），但是又没有完全锁死（后续滑动分散受力），这个度的把握还需要在实践中不断尝试。关于铅坠的松紧问题，有兴趣可以看看云贵地区常年用珠珠钓法的钓鱼人九哥:。

来源：钓鱼人九哥 - 做得不好仅供参考，高手不要喷我

总的来说，通过观察大量水下大鱼进食和钓鱼的视频，借鉴欧鲤钓和四川珠珠钓法，这一步刺鱼算是有一个好的办法了。

* 防挂底和真饵假饵

一般海竿钓底基本用的是串钩，因为钩子比较多，而且分散隔开有个几十厘米的距离，可以增加中鱼的可能性。但是串钩有根问题是中鱼之后溜鱼阶段容易挂底，也就是没有挂上鱼的钩子容易在鱼乱窜的过程中挂到其他障碍物，容易造成跑鱼。另外一个农业的禁渔政策也强调最好是单钩单线，禁止生产破坏性的钓鱼，一线多钩，一人多杆，爆炸钩、串钩和可视化锚鱼。就大水面钓大鱼而言，钩子多那么几个，其实能增加的中鱼的概率跟挂底的几率相比，作用可以说几乎是抵消了的。

用单钩确实可以减小溜鱼阶段挂底的可能性，但是抛竿和收杆仍然会有不小挂底的概率。平躺的钩子，容易陷入在石头缝隙里，同时如果淤泥水草多点的，钩子甚至直接陷入了淤泥里，根本看不到。要更好减小挂底率，也为了保证在淤泥水草多的地方钩子不躺底，可以借鉴上面提到的欧鲤鱼钓和珠珠钓的反底钓法，也是钩子不是平躺着的，是借助钩子上的丸饵或者塑料浮球将钩子带起来浮起来。

欧鲤钓主要还是在国外比较流行，简单说就是假饵浮球的反底钓。网上有各种评论欧鲤钓，其中常见的有一种：

一看就是外国视频，咱们国的鲤鱼不吃这套，咱们资源少[我想静静]

这个说法也确实有些道理，国外的鲤鱼特别大，是肥壮的那种，肥的壮的甚至有点带纹身鳞片黑黑的common/scaly那种，略微有点吓人（你可以在油管上搜一下carp fish，可以看到封面上各种肿胀不堪的鲤鱼，跟国内的比较苗条或者说也有大体的但是形状不会那么奇怪的肥），另外国外地广人稀，亚洲鲤鱼甚至一度成为了入侵物种，繁殖能力max, 密度肯定是比国内高，以我的经验看，在国内3-5斤以上的鱼就可以算是大个体鱼了。

那么是不是欧鲤钓就不值一看了？肯定也不是，研究背后的钓法逻辑，其实看到四川的珠珠钓法其实差不多，说明这个逻辑是成立的，起码是有值得学习借鉴之处的。这里说的不吃这套，主要是这个浮球假饵，因为要产生浮力，必然主要是塑料，这个问题就变成了：“我们人都不会吃塑料，鱼会吃塑料了？肯定是扯淡嘛“。但是看似理所当然的直觉，却可能只是人的惯性思维而已，到底是不是这样了，鱼是通过视觉、味觉还是什么的来判断可不可吃了，其实只要观察一下网上那些水下摄像的视频再结合四川地区流行的珠珠钓法的基本原理，就可以大致知道这个问题的答案：鱼会不会吃假饵问题不重要，但是鱼会有好奇心，尝试看看这个能不能吃，吃进去吐出来，只有这个动作就可以让钓鱼的逻辑成立。这里有一个珠珠钓法的视频（钓鱼这么久，我还是第一次听说不用饵就能钓鱼，上鱼全凭小珠子），欧鲤钓的（欧鲤钓不行？），还有人尝试过假饵+真饵，也有单独尝试过只有假饵，也有中鱼（当然这里要排除个体的因素，最好是有区域性的，这样从概率上才能一定证实这个钓法确实可行）。

实际上欧鲤钓里也有钓底躺平的，How Commercial Carp React to Your Bait Underwater!，这个在光滑的水底是不错的，看起来隐蔽性更高，在具体实践中，可以加多一个传统的挂玉米躺平的钩子试试看哪种效果好。

个人感觉和分析来看，在国内的野生水域(重点野生水域，不是斤塘黑塘），到处有电鱼网鱼猫鱼的，很少有鱼能长大特别大的个体，就鱼体的大小和密度而言，可能钩子和浮球稍微小点的，比如经过很多人校验改良过的四川的珠珠钓，可能更合适。或者将欧鲤钓的钩子和浮球搭配要用小尺寸的，用到最小的尺寸，钩子和浮球之间的距离要控制好，甚至可能最好是自己单独购买更小的钩和浮球制作调试钩组，降低目标鱼个体的预期，才可能达到好的效果。

这里还有一个常见的说法就是有人说了“我照着试了几次没有效果，这个钓法不适合纯属瞎搞”，个人觉得了，这个一个是没从概率论上想通，另外一个网上视频的渔获爆护这些东西让他的预期太大，导致一旦开始照猫画虎的效果不好，立刻就是失去了兴趣，于是用这个钓法不适合本地的借口让自己好受点。一开始照猫画虎没问题，但是一定慢慢要加入主动的思考总结调整，这样就意味着这个摸索期的不确定性是比较大的，在这个基础上要调整好心理预期和心态，这样才能在其中找到乐趣，否则什么钓法都白搭(No Silver Bullet – Essence and Accident in Software Engineering）。

欧鲤钓有几个关键，其中一个是钩子与浮球的大小搭配，需要保证大小角度合适，保证鱼能够比较丝滑的入口，否则比如浮球太大钩子太小，鱼很难吸入口或者吸入吐出时钩子太小挂到鱼的下嘴唇，一定是将将好；第二个是反底的高度：一般来2-5厘米比较好，如离底太高，好比鹤立鸡群，跟窝料不在一起，反而容易引起鱼的警觉；但是这个不是一定的，要根据什么来了？水底的情况，比如水底的湖床或者河床是不是硬底的，是否有淤泥或者水草，如果硬底可以1-3厘米，如果是淤泥水草，则可以调高一些。

所以了解水下的地形深浅非常有关系。

* 探索水底地形水深

海竿钓底一般都是没有浮漂的，那是不是代表就可以随意抛一杆，想打哪就打哪，就不管了？还真不是，我们需要提前了解好钓点附近的水下的地形水床的情况和水深。

了解水深很重要，大鱼每个季节栖息或者活动的水深不一样的，根据气压、风的大小、温度的变化会有相应的变化，结合化绍新的分享钓大鱼的水深这个月份多深合适？大致经验：

春钓 - 钓浅 3-5米

夏天： 20-30斤，6-7米； 8-10斤，5-6米；

中秋之后： 5-15米， 10-15比较多

一般来说4-8米深比较合适的。但是这里也要具体对待，比如冬天就不一定是钓的越深越好，最好是找有障碍物的比如芦苇草丛、树木、石头堆或者断崖坎等等结构，大鱼喜欢在这些比较安全的地方活动，水深不一定特别深。冬天白天可以钓远一点，但是晚上可以在近岸好点。太深没吃的，太浅不安全。另外比如冬天温度低，鱼是冷血动物，进食欲望低，就要适当用小钩子，窝料的大小要小点，比如把玉米颗粒掰成两半，窝料的量要稍微小点，散一点，就不是跟夏天一样越多越大越好，否则容易死窝。

Fox Carp Fishing Youtube: CARP FISHING IN WINTER (Catch more on cold days)

所以说一上来就随便乱抛，很有可能碰到那个钓点刚好有个凸起，水深只有一米或特别深，那有大鱼活动的可能性就太低了，也就很有可能会浪费时间和精力。

Talk is cheap, show me the code。有没有点切实的证据可以说明大鲤鱼在不同季节的温度、风速、气压下的活动轨迹，而不是凭经验或者自己的凭空推断了，毕竟年纪越大越感觉某些自以为是的经验靠不住-需要被像笛卡尔一样谨慎地审视？还真有，德国的一个渔业团队在一个自然水域的池塘湖泊里运用电子装备扫描追踪30-50条大鲤鱼在一年不同季节中的活动轨迹：

source来源: IFishMan Youtube: Wie verhalten sich Großkarpfen in einem natürlichen Gewässer? Hier der August

有兴趣的可以去这个频道仔细观察下不同鱼种（除了鲤鱼之外还有其他的鱼）的活动轨迹和习惯。从上面这些图片和频道上更加详细的视频观察到，就冬天而言，鱼不是在湖最深的西岸，而是在比较浅的东岸，不是一只两只，是聚集，还有喜欢在有一些障碍物和结构的地方活动（如果图片上没有点，就是因为鱼躲在了草丛或者树木或者石头下面，电子设备追踪不到，所以视频有时候只有几个圆圈，并没有看到所有的鱼）。

再一个是，最好选择深浅交界之处，摸清楚哪里是缓坡哪里是急坡，有没有水草挂住。测量水深，我们可以借鉴台钓的调漂的想法，加一个浮漂比如大肚漂用太空豆别针挂在主线八字环之上。这里也是有一些技巧的，首先固定大肚漂的档，最好不要用太空豆，因为一般30-50米之后的水深应该是会在三米之外，所以抛竿的时候太空豆需要穿过杆子最上面那个最小的导环，这个时候就容易挂住导环，导致出线不顺畅，甚至损坏，所以这里建议用更小更丝滑的棉线结作为太空豆的替代，借鉴矶竿钓组用 半圆挡片+棉线结或者棉线豆替换太空豆，这样减少阻力也更容易出线（棉线结是一个非常使用的小技巧，后面在精准抛竿中还会讲到）；再一个就是目测线长，铅坠到漂的这块线的长度来大致估算水深。最简单是目测，但是不方便不够特别准确，可以用油漆笔在主线的前端每隔一米做一个标记，因为油漆标记不容易溶于水，所以即使在水中浸泡也不会褪色，这样直接看几个标记就知道确切的水深。

测量水深的范围不要特别精确在同一个地点，相反应该是目标点位的周边，半径几米（3米）之内，上下左右看看水深来判断地形，多抛竿几次就可以了，观察是否有水草挂上来。

当然，关于鱼的习性，这些都是基于前人总结的经验或者小范围样本的数据分析，从一个怀疑论者来看，过去积累的这些经验是不可能适用所有将来场景，只能作为一个指导guideline. 正如李小龙Bruce Lee在龙争虎斗《Enter the Dragon》有个台词：

“Don’t think. FEEEEEEEEL! It’s like a finger pointing away to the moon. Do not concentrate on the finger or you will miss all of the heavenly glory!”

指月之指：

要理解截拳道,就像手指指向月亮一样,千万不要误将手指当成月亮,更不可因专注于手指而忽略天空其他美景。手指的作用,只是指引光明,至于你能获得多少,抑或眼界有多远,就全靠自己的领悟与努力了。

经验是有用的，给到一个思考的大概框架和切入点，但是不能当做金科玉律，盲信，具体问题还需要具体分析，结合实际情况，思考和灵活运用。所以了，哪怕你根据历史最佳最丰富经验和实际具体的观察，做出了最理性最合理的判断和策略，这个也只是你以为的，你不是鱼，你不知道鱼是怎么想的，不能把人的思维角度套到鱼的上面，期望鱼怎么活动，你无法把握鱼这个不确定性，这个不确定性才是钓鱼的这个乐趣所在，这个就跟股市投资一样。否则什么你都弄明白了，或许是过去的成功一度让你感觉你掌握了终极的规律，那这个还有什么乐趣所在？问题是：这可能吗？

应对这种不确定性，付出努力、做好准备、降低预期、开放心态、持续学习，在过程中寻找一种对自身自我和外界的好奇心。

* 精准抛竿

在准备阶段，抛竿使用的频率是比较多的，一个是打窝，一个是测量水深，两者都要求能比较精准的打到30-50米甚至更远之外的同一个点附近，不能偏差太多，但是海竿的抛竿不像手竿杆子离钓点的距离近那么容易精确地抛竿，针对海竿，那要如何能够做到精准地抛竿了？

首先精准是什么标准？怎么定义？

我们先看看打窝的需要，因为打窝不是要打到一个精确点（不像手竿），大水面的话，最好是粗中有细，细中有粗。就是说最好先打一个比较散的窝点，最好是以钓点为圆心，半径为3-5米的范围内，这个散窝的目的在于利用窝料的穿透力，先在大范围地吸引鱼，给鱼一个信号，这边有吃的；然后在这个圆圈的差不多的中间，打一个精准的小窝和我们的钩饵。大窝比较分散，中间的小窝比较密集，这样鱼进入大窝之后，慢慢的从周边吃，直到我们的小窝和钩饵。

怎么打一个半径几米的散窝了？这块可以借鉴下下面几个钓友的视频（海竿远投续窝有难度？教你一招搞定）、（海竿专用打窝器，钓鲤鱼必备，比买的更好用）、（野钓鲤鱼神器，钓鲤鱼如探囊取物，鱼竿差点被大鱼拉跑），用类似娃哈哈瓶子和翻版铅块，可以让海竿抛出去之后在快要到钓点时候窝料散落出来，形成一个大概三到五米的半径的区域。

我实际改装测试下了，我也买了个娃哈哈的瓶子，50克的铅坠，但是铅坠怎么也砸不扁，却意外发现椰树那个瓶子的底座刚刚好能够把AD钙奶覆盖住，效果出奇的好。

注意落水之前，窝料散开，形成一个直径1-3米的范围，多打几次，就可以形成一个比较大的散窝

自制简易打窝器：娃哈哈瓶子+铅坠+椰奶的底盖。忽略里面的蚕豆，用来打窝实验的，比玉米便宜。

有一个问题就是因为这个瓶子不是流线型的，空气阻力大，打不了没有瓶子那么远。本来直接打，可以打个30-40米远，带瓶子装上玉米只能打一个20-30米。我第一反应是杆子太短太软，毕竟拼多多的，想换更长更硬的，但是转念一想，钓近这个对于我这个钓鱼的场景而言根本就不是问题。因为这个自动化钓鱼，人不必要守着，所以钓鱼的时间从白天扩展到了黑夜。我看了很多国内外钓鱼的视频，却没有发现太多在后半夜钓鱼的经历（也就是从24：00-06：00）这段时间，甚至冬天的话，从晚上06：00到第二天早上06：00之间，很少有人会去钓鱼。一般都说，半夜天气冷，加上水草释放二氧化碳，从理论上看，鱼肯定是活动力不够，不会进食。但是，依据之前提到的，理性的怀疑审视经验，有种说法是进入十月份之后，鱼一般都是晚间活动,上半夜多余下半夜，结合上面德国的那个渔业实验，大鱼好像下半夜确实也有活动，并不是就是不动，再一个晚上安静，鱼反而会游到浅水岸边去寻找食物，所以综合这些经验分析来看，我非常好奇下半夜到底能不能钓到鱼，什么时间段什么温度变化曲线能钓到鱼，积累样本，去证实或证伪一些说法。所以这个时候20-30米远，没有那么远，反倒是一个好的策略，就不是什么问题了。

打小窝怎么办了？这里可以运用水溶网PVA，让打窝和放钓一次完成，诱钓结合。水溶网是一种材质，遇到水之后会在一断时间之后自行溶化化解，理想中我们希望它在铅坠到底的时候开始溶化，（水溶网根据温度水深压强的不同，溶化速度不一样，夏天快冬天慢，从几十秒到几个小时都有，下底过程一般是希望快速，所以重铅比较好），这样到底时水溶网里面的玉米就会破网而出，跟钩子能集中在同一块地方。同时我们还可以把钩子轻轻的钩在水溶网外面（中间紧绷的那块区域），这样可以防止钩子子线和主线八字环缠绕，一举两得。这里友情提示：水溶网一定要装满，而且一定要是满满鼓起来那种，然后拉紧不能松松垮垮，这样才能保证溶化的效果。

好多评论说水溶网融化太快，太垃圾，而且装不了湿饵，这个就是多此一举，其实了面对问题，对着淘宝卖家发泄差评、无能狂怒之后了，可以试试想想这东西是啥原理，有没有理解对，这样即使有问题不足，也许可以想其他办法绕过去。水溶网不同水温深度溶化速度不一样，稍微了解下水溶网溶化的原理，转个弯就行，溶化快因为网与水接触的太快，如果可以稍微隔绝下，就可以延长时间，另外做个试验就可以证实水溶网溶水但是不溶解油的，所以可以用比如芝麻油在网的表面涂上一层油，再不行，在面粉里裹两下，绝对可以延迟水与网接触的时间，延长水溶的时间，保证尽可能到底才化。装不了湿饵，是的，因为水溶网遇水就溶，直接装湿饵肯定有问题，但是可以把湿饵搞点面粉啥的稍微吸水一下，还可以在湿饵的表面涂点盐或者将湿饵浸泡在高点浓度的盐水之中再去装网，都可以保证水溶网的状态和正常使用。（老实说这些技巧淘宝商家应该写一份提示给到买家，这样买家不用走弯路，可见淘宝卖家其实也没有实战过，不了解钓鱼的实际应用场景和碰到的问题）

至此精确抛竿指的就是：能打出30-50米远的钓点，误差范围为3米左右。

这里选择30-50米，考虑到其实也不是越远越好（只是其中一个因素，水深地形也是很重要），越远收线时间更长，同时抛竿细微的变化导致几十米之外的误差放大的很厉害；另外矶竿可以轻松打出去70-100米远，因为他的线非常细，但是我们是钓大鱼要求线的强度拉力强，所以用的都是大力马编织线，6号甚至8号，9编的，这个必然会影响出线的效果；再一个，杆子和渔轮也会有很大的影响，主要是杆子，杆子越唱越硬，抛的越远，但是考虑到我们目前实验是经济实用型，拼夕夕型，所以杆子的质量要求不高，所以30-50米听起来是目前最好的选择；我目前是2.7米的杆子（20-30块），5000的轮子（塑料，20-30块），6号大力马主线，铅是100克的，抛出去大概也就30-50米左右，这里可以把铅属实太重了点，其实实验下下降到60-80克，应该可以抛的更远，后面再说。

如何确保尽可能精确的打到30-50米远之外同一个点了？这里细节就比较多，个人觉得有几点：

姿势 - 最好双脚迈开，并排，不要一前一后，看习惯，杆子位于脑袋正中央的上方
三点一线：找准对面远处的标记物，比如某一颗特别树木或者建筑物，以它为锚点Anchor, 然后杆子对准它
渔轮设置：锁死卸力、锁死卸力、锁死卸力，打开渔轮罩，食指勾住线（最好带手套，否则容易割伤手）
发力松手时机：发力不是用死力蛮力，而是用左手握住杆子根部那块，利用杠杆原理一样，顺势带下来；再一个食指松手的时间可以自己摸索一下，太早松开，容易打飞，也就是非常高但是不远，太晚松开，容易打的直，也不远
竿稍留出的长度：一般来说是杆子长度的三分之一或者二分之一，这样铅坠的惯性比较大，但是意味着杆子承受的压力也比较大，这块需要自己摸索
卡线：第一次打出去之后到了目标点之后，可以在渔轮这块线的地方，加一个棉线结或者将线卡在线杯卡上，这样后面再抛竿就能保证出线的长度是一样的，能落在同一个点附近
安全：一定要观察后面又没人，顶上有没有树或者电线，确保安全

这块还有其他因素，比如鱼线在线杯里是否绷的够紧，就跟其他钓鱼基本功一样，多多练习，尝试各种调整总结，慢慢就能熟能生巧。后面会发现，30-50米误差3米其实不是那么难的。

* 弹簧泄力

为了隐蔽性的要求，我们抛竿之后，会把竿子拿掉，把线直接拴在比如地插或者树根上。有过偷鱼偷钓经验的钓友就知道（比如这里8号的大力马线最大能钓多大的鱼啊，当然我这里不是偷钓，开放的)，这样一来有一个问题，就是有人值守的话，中鱼之后人可以用鱼竿和渔轮来帮助泄力，但是现在没有鱼竿和渔轮，直接就是线跟树桩或者地插硬对硬地硬怼，没有缓冲，这样是钢丝绳也说不好被拉断，更别说你是6号或者8号的线。十几斤的鱼，可能没有啥问题，如果是更大的，那估计就玄了。

根据电商项目的经验，流量直接打到底层数据库上，那谁来都不好使，一般的改造思路中有一点是层层分流，形成一个漏斗状的流量分配，这样可以让每一层都能恰当好处的发挥其最大优势。有鉴于此，我能想到是一个办法是多放点风线，也就是放出去多点长度的线，这个不像是普通下面的海竿，放出去之后会把线拉紧（因为不这样的话，无法保证能够拉动竿稍，出发铃铛或者报警器报警），我们反其道行之，多放点线，不需要太紧，这样一是可以避免鱼蹭线，第二假设中鱼之后，最差的情况鱼往反向逃窜，形成最不利的180度拔河，也可以有足够的线的余量，这就相当于多余的安全绳的，理论上来说鱼也不是无限制的往前冲；第二个办法是加上一个强力的弹簧，在主线和树根地插的连接处，提供一个缓冲，鱼线系在弹簧的一端，在绕过绑在地插或者树根上，这样可以在不影响隐蔽的前提下提供泄力。

当然我觉得最好的，还是可以在地插上有一个迷你的渔轮，因为这块的线不需要太长，只是为了大鱼第一下发力的的缓冲而已，但是最好是金属的，质量比较好，直接绑到主线上，然后调节好泄力，但是这个比较麻烦点。

* 中鱼检测装置

有了上面这些基础知识的积累和分析之后，接下来就是如何加上自动值守的装置，其实原理很简单。

ESP8266(微控制单元- MCU): 读取输入端角度或者加速度的变化，然后发送命令给到输出端(SIM800C)无线通信端。

MPU6050输入：检测鱼咬钩的动作，可以利用陀螺仪的原理，当鱼咬钩拉动鱼线和浮漂，产生一个z轴的加速度或者倾斜角度的变化，这个装置可以读取相应的值，并发送给MCU控制单元。

SIM800C输出：接受AT指令，可以插入手机卡，进行电话、短信和GPRS网络传输。简单来说，当中鱼了，MCU控制单元可以发送打电话指令给SIM800C，然后SIM800C可以执行打电话的操作，然后我的手机就有来电，知道已经中鱼。

当然这块实际比这个要复杂多点，比如如何供电，防水，所以基本上将这一块挪到了单独的一篇文章，进行详细描述。

总结

这一篇文章回顾和分析了海竿钓大鱼的技巧原理，根据我们需要达到的目的，自动化钓鱼，借鉴了哪些钓法，需要哪些调整，包括理论和实际的，从这些分析来看，可行性还是没有问题的。通过将大的问题，一步步拆分，每个小环节都进行理论和实际的实验，在理想完美性和现实中是否执行起来简单与否之间尽量找一个平衡，编程里有句话，不过过早优化，精益敏捷，fail fast。当然可以预见的是，野钓大鱼本来就是可与不可要求的事情，在这个不确定性基础上的实际调试将会是比较费时费力的，有效性还得经过实际检测才行，但是我们可以通过小范围模拟，比如小点池塘鱼缸，一步步调整推导到真实的领域，这中间肯定会有偏差和变化，但是完全是乐于见到的，也是过程乐趣的一部分。

下一篇文章将会讲到自动钓大鱼里这个核心装置的内容。

[钓鱼的感想(三): 自动钓大鱼]

参考

钓鱼的感想(一): 手竿钓大鱼

2021-10-01T04:55:32+00:00

最近几个月（5-8月）陆陆续续有点点时间空余，在老丈人家闲来无事，刚好旁边有个池塘，也是好久没有钓鱼了，于是想重温下小时候钓鱼的乐趣。池塘很小，估计半亩大小，离路边很近，旁边都是稻田，而且池塘中间长了一片水草占了一半的面积，还会随着风吹的方向不停的移动。

钓小鱼

小时候钓鱼很简单，山上竹子砍一根，菜地里蚯蚓搞一点（甚至尝试过去茅坑里挑出来蛆，白色虫子），线也不晓得是从哪里搞得，钩子有的还是自己搞铁丝弯一个钩的样子，就差不多可以钓了，那个时候都是钓白条和鲫鱼，也都是比较小的鱼，钓着还是蛮有趣的。

现在去搞竹竿那还是费事，所以去镇子上买了根70多块的4.5米的杆子，配了一个4号袖钩，0.8的子线， 1.5的主线，这个还是我那个之前喜欢钓鱼现在退了钓鱼烧的堂弟帮我一起去配的，这块比我小时候当时可还是讲究了不少。之后，他给我演示了下怎么发饵，我就兴冲冲去钓鱼了。不过这个商品饵料的调配我是不太清楚的，只虽然说相对蚯蚓干净了不少，但是这个饵水比、加多少拉丝粉，尝试了几次发现效果不怎么行，要么是出丝效果不行沾不上钩，要么太死雾化差点，在那里瞎忙活了半天。

ps：虽说也确实钓到了，但是没钓到多少 :(

兴冲冲钓了几次，被打击后，发现始终这个饵料的状态有点怪异，这才想起来去看看新手该如何发饵的教程。这下顺带是把钓鱼的基本流程梳理了一遍，发饵的饵水比最好一开始还是老老实实用量杯，虽然没有老手展示的信手拈来那么酷，如何削减铅皮配合浮漂来找底，酒米打窝等等。做到心里有底之后，再去去钓，果然效果是比之前好多了，一会就钓了不少.

钓小鱼虽然有意思，配合小竿细线确实有钓大鱼的那个爽快手感，白条油炸确实也很好吃，就是一个是抽的频率高手也有点累；后面有一次尝试夜钓，头上带上射灯，旁边点着蚊香，在满是草的堤岸上钓了几个小时，虽然看的是眼睛很不舒服（连白天看漂都有点累，更别说晚上），倒是也上了一两条一斤左右的大板鲫（虽然后续尝试过夜光漂，但是眼睛太累了，另外从安全考虑，夏天晚上蛇出来的几率高，所以基本后续不在夜钓）。

钓大鱼

老丈人看到钓到了大个体的鲫鱼，说道上次有个亲戚在里面钓了一个三斤左右的鲢鳙，里面有大鱼，你可以试试钓草鱼试试。这个正好勾起了我想挑战大鱼的兴趣了。堂弟建议钓草鱼，让我买点老谭玉米打窝，然后买个新鲜玉米钓，买个大点的七号袖钩。于是大热天的上午，先是打了两三勺子的老谭玉米，然后挂了新鲜玉米钓，怎料一上午没口，反倒是皮肤晒得跟非洲人似得。下面是我几次钓鱼跑鱼（哦豁）的初体验: （跑到的总是最大的，我估计其中至少有一条百来斤 :)

ps: 这是后面水库坐钓时候拍的，基本上垃圾我自己带来的都会带走，但是旁边还是有很多其他钓友丢的垃圾，比如左上角的方便面盒子，钓鱼还是蛮拼，但是垃圾还是自己最好带走。

* 第一次跑鱼 - 主线太细

中午一直到四点半天气太热了，是没有办法钓鱼的，到了稍微凉快点，接着坐钓，这次新鲜玉米打窝新鲜玉米挂耳钓。等了后一阵子，看到了零星的“鱼星”（底下有鱼聚集的迹象），过了一会，突然一个黑漂，给我一个激动哦，直接一把提竿，立马手上感觉这个鱼的劲头很大，还没等我准备使劲拽过来，突然一下，这个力没了，我一看主线断了。

经过这个之后，老丈人说这里面的草鱼估计至少也有3斤也上，可能5-10斤都有，你这个主线和钩子都得升级，得用大钩粗线。于是乎看了看别人是如何钓草鱼的、草鱼的线组，在拼多多上面买了4.0的主线，2.5的子线+7号伊势尼的线钩搭配，一切以便宜实用为主。

* 第二次跑鱼 - 鱼竿太脆

线组到了之后，我迫不及待的去塘里去钓鱼，上午顶着大太阳都没口，也是等到下午快到傍晚的时候，突然来口了，漂相猛地一沉，我赶紧提竿刺鱼，这次第一下没有问题，但是这个鱼马上往左右边跑，我就朝着反方向杨杆，企图把它拉回来，结果一声清脆的声音，杆子断了。

根据这个断像来看，感觉这个品牌也应该是杂牌子，70块的鱼竿，还是质量不行啊。问题是我这也只有这一只竿子，只能去堂弟那里问问他后面不钓鱼之后钓具还在不在，借过来试试。他刚好有一根5米4的杆子之前用来水库里钓大鱼的，于是就拿过用，对付8-10斤以下的草鱼应该是没问题的。

* 第三次跑鱼 - 子线八字环绑的不牢靠

这次换了大线大杆，虽然线短（4.5），杆子长点（5.4），但是问题不大，这次应该手拿把攥，跑不了吧。这次上午快10点时候，一般来说这个时候天气特别热，应该不是草鱼会开口的时候，我都快放弃了，却发现有了不少的鱼星。第一次次没有让鱼跑掉，杆子的这个腰力特别不错，让我发力也比较有信心，确实被我直接没怎么溜，拉到了岸边，自己伸脚从堤岸边上下去一把鱼按住，赶紧给扔到了旁边的水稻田里。是的，我这是乞丐式钓法，没有伞、没有鱼护、没有抄网、没有钓椅。后面也还上了一条，但是跑了起码四条，无一例外都是提竿溜鱼，鱼往水草里跑，或者往前方跑，突然子线断了，双钩脱了一个，鱼跑了，这个感觉真是大腿都拍青了。

应该目测一个有3斤，一个有5斤左右。

第一感觉是不是这个拼多多产品不行，这个是拼夕夕，子线和钩的质量问题。但是回头我又琢磨了，为什么都是子线断了，而不是其他地方断了或者脱钩，比如主线、比如子线和钩柄那里。回头我跟堂弟聊了下，他经验丰富，问我怎么绑线的，特别是八字环和子线那里。我说我就是按照之前钓小鱼的绑法，类似主线跟竿稍固定的方式，直接绕过来穿过八字环，然后把钩放过来，类似

他说你这个方法钓小鱼可以，但是钓大鱼不行，不牢靠太松了，大鱼瞬间施加的拉力很大，整个线组会绷成一条直线，会在最松最不受力的地方先断掉。他建议看看网上视频关于如何绑线的教程比较直观点。下面是调整了下的八字环和子线的绑法：

主线和八字环类似：

可以多绕机圈，子线比如3-4圈，主线6-8圈，最后在规整和拉紧线，这样当中鱼受力的时候，在这个结这里的拉力，只会和八字环越拉越紧，保证能够更好受力。

第n次跑鱼 - 杨杆刺鱼、满杆、溜鱼不当

修正了上面那个线的绑法之后，后面几次钓鱼中鱼之后基本不会出现上面那种断法，但是在子线和钩柄那块容易塌、形成拔河，跑了几次。一开始还是觉得这个质量有问题，后面跟老钓友聊了下，他们说你杨杆刺鱼的力量太大了，年轻人太激动力量控制不好（可不是，等了大半天好不容易来一口大黑漂，那个激动哦），刺鱼的力量太大了，基本上差不多就行。这一步看起来是准备、打窝，坐钓、中鱼、到了溜鱼最后一步了，这一步特别容易激动还有心理上比较松懈，其实这个环节是非常重要需要冷静处理的，急不得。杨杆刺鱼的时候，不能力量太大，最好可以稍微感受出鱼的力量和大小，同时溜鱼也是有技巧的，不同急于上岸，直接往岸边飞，这样要么杆子断要么线断。最好是慢慢的溜，溜的它没力气，这样上岸的取钩都能更稳妥安全点。

之所以形成拔河，也就是上图右边这种常见的，一个主要问题：是线的余量不够，杆子要么一开始就没有很好的站立起来，这样无法利用杆的腰力来卸掉鱼的冲击力，所以这个时候线承担了几乎所有的拉力（杆子无法护线）。

解决这个问题基本上可以试试：1.别抛满杆。

虽说一寸远一寸强，钓的鱼体型能更大，但是钓鱼这个我后面总结经验来看（爆炸钩、窜钩挂底），跟打战和炒股是一个道理，在进攻的时候首先要想到防守，毕竟客观条件的限制在那里而且还有主观上的心理偏见（参考查理芒格的《人类误判心理学》）在那里，所以哪怕杆子线组在厉害，也要有余量。所以一个办法是杆子抛出去了，往回拉点，一个是线组长一点，比杆子长个几米，这个需要根据水深目标鱼大小，或者搞一个安全绳，这样线组也算是变长了。但是基础这块，杆子的腰力调性，包括线组的搭配，肯定也是一开始就摸清楚的。

溜鱼时技巧。

最懂还是尽量避免杆子没立起来，赶线鱼成为一条直线，合理的分散的力的传导。如果鱼往前冲，杆子还没立起来，除非有安全绳，基本上这个是最容易出现的跑鱼的；鱼往左边跑，要么人跟着左边跑，要么杆子往右边倒，但是切记让杆子线和鱼成为一条直线，最好是有一定的夹角，可以让鱼的力量在这个角度上面分散掉一部分力量；注意旁边的水草，特别是一般我们用的双钩，鱼一旦往草里面跑，另一个钩就特别容易挂在上面，突然一下子力量松了，鱼力量的惯性一下子就挣脱了把子线给扯断了。所以真正钓大鱼还是得单线通钩，一个钩，这样溜鱼等等不容易挂底，想想看好不容易守到大鱼，结果在快到留到岸边的时候给挂底跑了，这个酸爽哦。

当我回头总结这些经验，并运用的实际之后，感觉中鱼和溜鱼的过程明显不那么容易跑鱼了，钓一条三斤的小青鱼和八斤的草鱼。

好景不长啊，过了阵子池塘因为夏天天气太热，就是塘了有一半面积的漂浮的水草，导致有几天溶氧量不足，草鱼青鱼缺氧，几乎都死光了。无奈我直就只能转战旁边的水库，汨罗江因为长江支流限钓。

大水面水库

水库的面积还是蛮大的，也没有人承包放养鱼，基本上是一个自然的水库环境，水面广但是鱼的密度比较低，一边是山坡切下去-深水，另外一边是平缓点的稻田-浅水，基本上村民也都是在深水这边钓。我过去一开始拿玉米钓是半天都没口，后来问了下，长期在这里坐钓的村民，他们说这个这里很难钓到鱼，要待在一个钓点（所以基本钓位钓具都不拿走的）连续几天不断打窝打重窝，钓一钓鲤鱼翘嘴，运气好还是能搞两条。

我那天坐在那里跟他们一起钓鱼，一个是拿着十米的杆子，一个一根海竿，只看到拿手竿每隔一段时间，就打点玉米下去，补窝（为了留住鱼），但是一下午也没看到他们钓到啥，甚至口都没有，还白抽了一包烟。

这才知道钓野钓的真实的自然水域特别是大水域，跟我在池塘里钓鱼那真是两码事，之前的积累的自信荡然无存啊。总结下钓大鱼的理论和经验就是：

钓位选择，水得比较深： - 大鱼一般栖息在深水区，但是也需要在浅水区来觅食，所以一般深浅交接最好，总体来说深水容易出大鱼，但是也别过深。钓位的选择是有讲究的，当然跟季节、水温、气压（秋冬天钓的深一点）、目标鱼也有关系，水深水浅、是否是腐质、钓点附近的地区是缓还是陡、水下地形如何，需要先摸个透，一个是可以跟熟悉环境的当地钓友聊；一个借助比如卫星地图那种俯视图，或者坐钓前走路开车先转一圈，观察水库整个大的地形地貌轮廓，做到心中有底。
重窝子： - 要想鱼上岸，还得粮食换。得打重窝子，还得是连续几天不停的打，然后再同一个地方坐钓。要想在这么大的水域，把鱼吸引过来，得让它知道这个地方有吃的，还得是持续不断的喂养，让鱼知道这里定期有吃的，这样比较容易聚鱼。窝子最好是有穿透力诱力的饵料，比如老谭玉米那种酒糟浸泡过的等等，让其他地方的鱼通过嗅觉过来，不然你说人家远在上海大都市的鱼如何知道你这新疆吐鲁番这地的。我看到旁边的钓友有的是几十斤的往里面倒，当然跟李大毛-外号填湖真人-打窝还是差了点的。
晚上钓：水深藏大鱼，大鱼之所为大鱼，是有原因的。鱼作为动物的自我保存来说，只关心本能的两点：有没有吃的，有没有危险。危险的话，包括噪音，此处有没有威胁，有的话是否可以够快速游到深水区躲避；吃的话，这个东西能不能吃，我先试试，如果看看大鱼在水下吃食的视频话，会发现它们会试探性从窝子的边上零星散落的开始，而不是直奔窝料中间最多的饵料区域，吃一口吐出来，游走；然后又游回来，接着重复，知道确认安全才会吃；深水一般在中间，并没有浅水岸边那么常见芦苇或者落叶等等事物那么多，所以还得去浅水区寻找吃的。但是白天一般噪音比较大，晚上会相对小一些，大鱼这个时候会靠边觅食，所以手竿的话，晚上也是一个钓大鱼的好的时间段。
守：钓大鱼不必钓小鱼白条那么多口，基本上是半天没口。有句笑话说的好钓鱼只有两口，厕所和吃饭，要么点火抽烟，突然来一口把你的杆子说不定都拉走。大部分时间，比如你钓玉米或者螺丝，基本上都是在等，苦等，无聊的漫长的等待。不过经历背这沉重的钓具爬山涉水到达钓位，精心的钓前准备打窝线组搭配调漂，钓中那漫长枯燥的等待之后来的那一口，那个中鱼的快感，那个手感，那个风险拉的呲呲响的声音简直堪比天籁之音，那种无可比拟的畅快，这个也是钓小鱼不能体会到的。

基于手竿钓大鱼，你需要时间、精力和财力都得有才行，而且钓的地方还不一定很深（因为离岸边近），而且注意力的需要经常在浮漂上，总体而言还是挺麻烦的。有没有简单的点了？可以钓更远的距离，不用一直盯着浮漂，还可以不像手竿那么样需要经常抛竿补窝稍微轻松点了？有，可以用海竿。海竿有一个纺车轮，里面可以用来装鱼线，还可以调剂卸力控制出线的力道，溜鱼时相当于有了失手绳的作用，这个渔轮配合稍微重点的铅坠，可以轻松打出去30-50米、50-60米，甚至百米。一般还在竿稍那里挂一个铃铛，这样当中鱼之后，鱼猛然的拉力会带动竿稍震动摆动，铃铛会发出清脆的声音，你就知道中鱼了。这样就可以搞一把折叠床躺椅放在旁边，睡觉玩手机都可以了，只要听到声音去提竿溜鱼就行了。

海竿钓大鱼看起来比较轻松，甚至一般都是摆四根，八根那种海竿阵来提高中鱼的机会。但是海竿也有问题，你需要守着这些鱼竿，人虽然可以躺着休息，但是这回不是盯着浮漂了，得盯着竿子了，比如晚上你就得收回去，不然会被别人拿去。比如下面这个视频里，海竿在晚上自己睡觉的时候，被偷了，损失了不少（钓大鱼一般海竿和轮子比较贵）。

所以村子里有人海竿，但是不用杆子，用的是那个筒子，比如矿泉水瓶或者竹子的一节，把线绕在上面，然后扔出去之后（当然距离没有太远，但是也比手竿远点），把线的另外一端绑在岸边的的树根上或者石头上，这样一来旁边的人就不像杆子一样容易被发现，第二天定点去检查一下有没有中鱼，然后换个饵料接着扔出去等就行了。这个办法好点，就是需要每天定时去检查，另外手可以像八卦轮那样扔出去但是距离不远不准确，不好打窝，村子里也有人这样钓到了二十多斤的大青鱼，但是大部分时间都是碰运气，很少。

总结

钓鱼还是一项比较有意思的活动，大概率因为就像野钓你不知道你下一杆会重什么鱼，哪怕你用的同一个配方同一个程序标准，在同一个地点，昨天和今天钓到鱼也会不一样，鱼情也不会是一样的，这也是这种未知不确定性带来的挑战和乐趣，钓鱼圈子里有一句是：钓无定律，这个就形容的很好。

影响钓鱼的因素也是非常多的，比如饵料、天气、季节、杆子，线组、水文条件等等，这里面有没有一种因素是万能的，用了这种技法或者饵料就可以保证钓到鱼。而新手在接触新的领域，一开始最容易上手的还是视频这种形式，相对文字而已，这个视频这种直观连续的展示形式最贴合也是最符合学习一开始的模仿阶段。跟着视频手把手做，钓到了鱼，肯定对于初学的人来说是一个很大的鼓舞，就好比学习新的程序语言或者框架，你先按照getting started先跟着步骤搭建环境跑起来hello world，跑起来了后续再去结合实际看它的文档原理，回到钓鱼，视频里老手大师用的什么饵料，不自觉的也会成为临摹的对象。饵料不同品牌可能有区别，但是对于新手而言，之间的差距，没有实际经验的累积还不足以识别和准确的运用发挥对应的最大潜力，差距是不大的，所以不是说你新手就得买xxx品牌的杆子xxx品牌的饵料，就一定能钓到鱼。这个不就是伪先知了么？

新手建议还是跟我一样用便宜的杆子一般饵料，可以方便快速小成本试错，上手测试，这样可以快速掌握要门。否则买了贵的杆子和饵料，结果啥都没钓到，仅有的哪点热情马上就被现实捶打了，浪费钱还费心力。打个比方，池塘里草鱼不是死光了么，我就是想钓一下天的鱼，正好我老婆要不你学学邓刚钓鲢鳙吧，这塘早上她还看到鲢鳙浮到水面上张着大嘴换气了。完了我就看了邓刚钓鲢鳙的视频，去渔具店买了天元的饵料，接着就上塘实操，结果了虽然我的水比也是按照他视频里说的来，但是总是感觉要么是钩子贴不住，放到水里一下子就钓了，浮漂就上来了，要么是半天浮漂不起来，太紧了，捏的块太大，把握不住水比，因为塘不是那种水质清晰的，我就无法看到实际饵料的状态，搞了半天无功而返。后面回来我就老琢磨这个饵料的状态问题，水下的情况真是摸黑抓瞎，于是我找了一个小的杯子，用桶装了一桶水，按照不同的水比1:0.5, 1:0.6, 1:0.7 - 1:1.0, 1:1小量的是尝试每一个比列的饵料，然后观察在桶里它的附钩性和在水中的雾化效果，再试试摇动桶看看在浮动情况下的状态。水比背后的关键在于调整附钩和雾化，别人说的是别人经验总结的理论，但是我还得自己去实际体验才行。经过这个测试，这下就非常清楚的掌握了不同配比的状态和持续时间的区别，哪个比列比较合适初期打窝（雾化快），那些适合后期鱼来了坐钓（雾化慢）；天元的饵料还是太贵了，后面我换了其他便宜的饵料，也是不急于坐钓，先取小部分做一个实验，这样就知道在当时的气候温度下，饵料的状态大概什么是比较好的，结果上鱼的效果跟天元的没有什么区别（当然这是我个体的感受，也许我还不是高手，没有发挥出饵对应的全部的潜力）。这个也适用于比如调漂啊等等，可以在家里那个桶先试试，看看水下不同的情形下，浮漂是什么样子的，观察两个钩子的形态。

可以看得出来钓鱼虽然充满未知的不确定的，但是也不是毫无规律可循。钓无定律，并不是随机的漫无目的的，尽管充满不确定，变动不迁，有点像柏拉图的那个洞穴里的投影，变幻莫测，我们无法掌握水下面鱼的每个时刻的游泳进食的情况，但是还是可以像亚里士多德一样观察、思考、总结这个投影现象背后的规律和规则，找到自然稳定的秩序，不断试验学习，探寻体验这个钓鱼的过程和乐趣。这些总结规律就像进化论一样不是一朝一夕的改变，而是漫长的演化，所以在人短暂的存在来看，这些现象体现出来的规律是可以依靠，可以作为一些科学的原则和基准，从上面稳步的搭建科学的体系，尽管我们可能永远不知道那个背后的物自体。就像牛顿的万有引力，突破了传统了力是通过直接作用的传导机械式链条才能发生作用一样，不必依赖直接接触而产生的力，引力，但是牛顿终其一生无法理解和解释什么是引力，他只是观察到了这个力的表现和作用，推导出了这个规律和公式，这不妨碍这个全新的突破给人类科学发展带来的成就。所以总结思考，理论和实际结合，得到这些规律，还是非常有帮助和必要的，比如各种鱼的习性、什么样的饵料比较适合、水深水浅、跑铅定铅、走水等等，结合别人前任的经验思考的总结，摸索出一套自己的钓鱼心得。陀思妥耶夫斯基说的 To go wrong in one's own way is better than to go right in someone else's，这个试错过程中犯错，甚至犯很蠢的错误，是非常自然的，退一步说如果有一阵子你没有犯错，那表明很有可能你最近可能没什么长进了，没学啥新的东西；但是错误的蠢也要蠢的有理由，有根据，有条理，能追溯能拆解，能从中学习点东西，不然是真蠢，因为那个蠢是无法琢磨的，必然的，我们尽可能需要的是：有确定性的蠢有反思的蠢。

但是同时也要预见到这种经验规律的总结并非是不变的精确反应了实际的背后的未知，所以也要保持一定的开放性，需要有意识的检视下，正如康德说*休谟*的怀疑论打破了自己的独断论迷思，起码你知道任何宣传有万能的xxxx的，那都是骗钱的。另一方面了，也提醒自己如果做不到像维特根斯坦那样对于语言局限的认识，跟别人聊起钓鱼时，尽量别把话说满了，只敢谦虚的说交流一下经验，千万别说我的钓法比你强，天下第一。这样看来钓无定律，并非是随意随性的自由，而是建立在对于事物必然性有了深刻的认识之上的。

有句话说了，钓鱼中有人为鱼，有人为渔，有人为娱，有人喂鱼。还有一种不为啥，只是体验，各有各的玩法和乐趣所在，但是客观的来说也存在这鄙视链，钓路亚的鄙视手竿，因为手竿坐那就一天没有啥运动、久坐成疾、没有肉食鱼凶猛，手竿的鄙视海竿没有技术含量，手竿里面了又有不用安全绳鄙视用安全绳-认为用了安全绳就没了灵魂，溜鱼的鄙视飞鱼的，用竹竿鄙视用商品杆的，用钢筋的鄙视用竹竿的，野钓的鄙视钓黑塘的，不一而足。台钓典型的鄙视海钓的一种说法是：你这个钓法没有灵魂，钓鱼主要是在鱼吃饵的时候，抓准时机上钩，溜鱼跟鱼搏斗。

这个说法也有一定的道理，但是还是需要看看具体场景，不可以一概而论。如果是钓大鱼，特别是大的自然水域的野钓, 正如上面所讲，比较稀的大鱼（被锚鱼药鱼炸鱼轮番轰炸，野外鱼能长大个体实属不易）决定了你绝大部分时间都是白等，漂有如定海神针，纹丝不动，一两天甚至一个星期都没一口，除非你有充足的时间（退休了，否则996社畜哪有这么多时间），大部分时间都不是看漂而是玩手机去了，因为太无聊了，这样的钓鱼的灵魂又在哪里了？如果是为了那一口，那个鱼的冲劲手感，那为什么不选择细杆细线钓小鱼了，同样的手感，甚至还更带劲，钓法的变化线组的变化还更多，频率还高。

俗话说的好，蛇有蛇路鼠有鼠道，条条大道通罗马，钓鱼背后的不确定性和复杂性，决定了没有一种固定的套路是适用所有场景的，也就决定了没有一种终极的价值衡量标准来衡量哪种钓法更牛批更有灵魂 - 如果有，那只不过是你认为的。

钓鱼，个人觉得跟股市金融有点像，有些跟我们日常常识相违背的地方，这种不确定性，意味着需要用概率的角度来思考问题。这种思维的不同角度有时候能让我们在已经树立起来日常生活或者惯性思维中-寻求确定性- 跳脱出来，给人一些启发，检视这些已经深埋于我们潜意识里的习以为常的认知方式。哪怕你做好的最好的准备，万无一失，也有可能钓不到鱼，所以空军其实非常正常的，甚至可能是一种常态。家里人问我经常看到你背跟杆子出去，但是回来没看到一条鱼，真是搞空鬼列。要永远不空军，有没有办法？也有，钓鱼空军回来路上背头小牛、或者菜市场买条鱼对吧，钓鱼老永不空军。当你明白这就是一个概率的游戏，做足了准备，也有可能钓不到鱼，首先心态上就自由了很多，不急于下杆或者干啥干啥都急匆匆的，你也会更加容易察觉到钓鱼的那个氛围，你身边的自然环境，钓鱼特别是野钓，一般自然环境是非常棒的，特别是对于大城市的人，这是一种沉浸式的体验，一种跟山水自然的连接，跟自己的原始状态的连接。跑了鱼你的心情也不会太过纠结，或者说不会让它容易深沉到下意识里，不用留着郁闷影响到接下来比如跟家人吃饭的心情等等，跑鱼跟中鱼没有是么区别，有了中鱼的手感畅快感可以挺好，但是跑了鱼的心里默喊的那声哦豁也郁闷的挺畅快，接着后面在去研究分析跑鱼的原因罢了嘛。关于中鱼跑鱼，和钓鱼时做的选择，后面反悔遗憾当初选择（比如子线该用大点），自然联想到王弼的《物不迁论》，王弼关于运动和静止的探讨，日常说的运动，其实是静止，就像电影的胶卷，每一刻都是一帧，但是如果我们按照一定的频率播放，足以超过人眼睛的“余晖效应”也就是100ms一帧还是多少来着，人会意识到这是运动的，但是实际每一刻都是静止的。过去的物体到不了现在，现在的回不到过去，结合我浅薄的佛学的理解，不昧因果，过去的状态1（因）到现在的状态2（果），站在过去的状态1当时有无数条路可以选择，但是他只能选择一条，进入到现在也就是状态2；但是你不禁想如果我当初选了另外的条路，那个是什么状态，会不会比现在好？大部分情况，鉴于人的心里误判认知偏见生成本能，没有得到的比得到的好，于是有些唉声叹气，甚至不满意的，容易沉浸在这个“如果”里来彰显他们现在无力的原因不是他的问题，也许是但是无济于事。按照佛学来解释，你是自由的，当时你有很多选择，但是你又是必然的，因到果，其他没有选择的不过是虚幻的而已，你自由的选择了必然，听起来有些阿Q的味道，但是你心里明白当前只能总结过去走好下一步，该干嘛时干嘛。中鱼时就干中鱼的事，感受中鱼的快感等情绪；跑鱼时就是干跑鱼的事，哦豁肯定是会有郁闷的情绪，感受它，体验它，但是好了，这个时刻过去了，现在这个时刻了怎么干了，如果中鱼那就是表示方法对思路正确接着干，如果跑鱼就思考为什么跑鱼怎么调整，这样而已嘛。正如《晋书·王徽之传》的记载，东晋王徽之半夜兴起划船拜访戴安道，结果天亮到了门前却没有敲门进去，直接回去了。

人问其故，徽子曰：“本乘兴而来，兴尽而归，何必见安道邪？”

钓鱼为渔，非鱼；正如无产阶级战士在解放全人类的过程中也解放了自己。

用概率的思维去看钓鱼，就不会那么执着于结果的确定性，特别是个列、样本不够时，比如当你尝试新的钓法，做出不同的调整，就跟吃鸡的主播从习惯的比如二指改三指、三指改四指到六指一样，一开始肯定是有一个适应期，不见得立马你就得得到预期的结果；但是你不会轻易的由于一两次结果的不同就给这种新的方法下确定的结论：如果钓的好，可能是新手礼包了，就吹嘘这种钓法的无敌，相反钓的结果不好，就认为这种钓法不值一文，不屑一顾，去钓鱼的视频网站论坛都可以看到这种评论。概率讲究一个样本的大小，特别是钓鱼还是这种资源决定性占比较大的运动(主要是鱼的密度，差不多是7分运气3分技术)。

这个也是我最近这些年一直思考的问题，年纪是慢慢变大了，身体的技能明显是下降的，精神层次上的愉快满足感肯定不会像年轻时那么容易获得了，时间也没有那么多了，看起来事情也多了，但是相对于我年轻时，我现在的状态或者将来的状态就是没有办法通过其他的方式得到弥补么？既然精力的广度无法扩展，只能在深度上挖掘了，那就是提高品质和质量。正如上面所说，并不是求多，或者越多线程越好，相反反而是朝单线程的方向发展。也就是专注，吃饭时吃饭，睡觉时睡觉，钓鱼时钓鱼（钓鱼中间我是不会看手机的），沉浸式体验，对于过程的体验感受，超过对于结果的执着。这并不代表我是彻底的转向神秘主义，相反，正如上面钓无定律一样，已有的经验和认知积累不是虚无不是无用的，可以加以利用，同时需要强大的和不断学习吸收的体系做支撑，这个体系和过程的我也是一直摸索，提高自我的觉察力，更好了解自己，开放的扩展自己的能力，希望可以在时间精力有限的外在限制下，发挥最大的潜力效力功用，得到最大程度的一种体验。

扩展

上面提到海竿一般都还是爆炸钩和长沟，容易刮底，溜鱼时候容易跑鱼，而且海竿的杆子太显眼，容易被别人偷走，换成筒筒，方便是方便了，但是不好仍，还有一个就是还得是被动的等，检查有没有鱼。

那么有没有一种办法可以利用海竿的优势，钓远处、钓大鱼、而且还可以实现筒筒的隐蔽性，同时可以将那个竿稍铃铛换成一个装置，可以识别出中鱼的信号（比如竿稍或者鱼线的抖动），发送无线信号到我的手机上？这样就可以化被动为主动，我可以把装置设置好，抛竿，然后回家该干嘛干嘛，等到手机收到中鱼短信，过去直接拿鱼溜鱼，岂不美哉？

这是下一篇文章将会讲到的，隐蔽性的自动海竿钓大鱼。

[钓鱼的感想(二): 海竿钓大鱼]

REFERENCES

2020年关于现代经济的一些思考(六)

2021-02-06T04:55:32+00:00

引用References

Chairman Bernanke’s College Lecture Series: https://www.federalreserve.gov/aboutthefed/educational-tools/chairmans-lecture-series-about.htm
BOJ Becomes Biggest Japan Stock Owner With $434 Billion Hoard: https://www.bloomberg.com/news/articles/2020-12-06/boj-becomes-biggest-japan-stock-owner-with-434-billion-hoard
When the Federal Reserve Buys ETFs, Thank the Bank of Japan: https://www.bloomberg.com/news/articles/2020-04-06/when-the-federal-reserve-buys-etfs-thank-the-bank-of-japan
Postscript: Paul Samuelson https://www.newyorker.com/news/john-cassidy/postscript-paul-samuelson
Liquidity Traps- Macroeconomics https://www.youtube.com/watch?v=p47uvsjB5E0&ab_channel=JacobClifford
High Yield Bond: https://www.investopedia.com/terms/h/high_yield_bond.asp
Ben Bernake - Remarks on Class Day 2008: https://www.federalreserve.gov/newsevents/speech/bernanke20080604a.htm
How the Federal Reserve Was Formed: https://www.investopedia.com/articles/economics/08/federal-reserve.asp
Who owns the federal reserve: https://www.thebalance.com/who-owns-the-federal-reserve-3305974
What Is the Dodd-Frank Wall Street Reform Act? https://www.thebalance.com/dodd-frank-wall-street-reform-act-3305688
PROMISSORY NOTES ACT OF 1704 (ENGLAND) https://encyclopedia-of-money.blogspot.com/2011/10/promissory-notes-act-of-1704-england.html
A history of the British banknote
黄金货币帝国：掌控全球 https://zhuanlan.zhihu.com/p/61559210
建造美国的人 https://www.gcores.com/articles/20669
The Panic of 1907: J.P. Morgan and the Money Trust https://www.stlouisfed.org/education/the-panic-of-1907-jp-morgan-and-the-money-trust
The Real Villain Behind Our New Gilded Age https://www.nytimes.com/2018/05/01/opinion/monopoly-power-new-gilded-age.html
J.P. Morgan’s Character Lesson: https://www.bloomberg.com/news/articles/2002-06-06/j-dot-p-dot-morgans-character-lesson
J.P. Morgan Documentary: How One Man Financed America https://www.youtube.com/watch?v=5jjdErDkDZE&t=15s
https://www.investopedia.com/terms/f/federal-reserve-bank-of-stlouis.asp
https://www.investopedia.com/terms/o/openmarketoperations.asp
https://www.investopedia.com/terms/f/federal-reserve-regulations.asp
The 1870-1914 Gold Standard: The Most Perfect One Ever Created: https://www.forbes.com/sites/nathanlewis/2013/01/03/the-1870-1914-gold-standard-the-most-perfect-one-ever-created/
Economic slump https://www.nationalarchives.gov.uk/cabinetpapers/themes/economic-slump.htm
How the Bank of England abandoned the gold standard https://www.telegraph.co.uk/finance/commodities/11330611/How-the-Bank-of-England-abandoned-the-gold-standard.html
DW - 以史为鉴：贸易保护法案催化大萧条
花呗，借呗年利率到底有多高？
War Bonds explained How children helped pay for both world wars
Promotion of war bonds in the United States during World War II. HD Stock Footage
How Lyndon Johnson unmade the Democratic Party: https://www.washingtonpost.com/news/made-by-history/wp/2018/03/30/how-lyndon-johnson-unmade-the-democratic-party/
https://www.history.com/topics/1960s/1960s-history
Brief History of the Gold Standard in the United States
Richard Nixons Economic Polices: https://www.thebalance.com/president-richard-m-nixon-s-economic-policies-3305562#citation-16
Nixon’s Wage and Price Controls: https://www.econlib.org/archives/2016/12/nixons_wage_and.html
What is stagflation and why is it so dangerous? https://money.howstuffworks.com/stagflation.htm
Paul Volcker, the most important Fed chair of the 20th century, has died https://www.vox.com/policy-and-politics/2019/12/9/21002682/paul-volcker-dies-federal-reserve-volcker-rule
https://www.nytimes.com/2019/12/09/business/paul-a-volcker-dead.html
The Chrysler Bailout of 1979: A Retrospective https://www.investopedia.com/articles/economics/chrysler-bailout.asp
Spitting Image Ronald Reagan BritBox https://www.youtube.com/watch?v=keP17ZU6_RU&ab_channel=BritBox
How does Donald (Trump) compare to Ronald (Reagan)? - BBC Newsnight https://www.youtube.com/watch?v=C_lY_XD6aVA&ab_channel=BBCNewsnight
How tax brackets actually work https://www.youtube.com/watch/VJhsjUPDulw
Who pays the lowest taxes in the US? https://www.youtube.com/watch?v=kXCGbAv8YPw&ab_channel=Vox
https://www.vox.com/videos/2019/3/5/18249732/wealth-tax-better-way-rich
The Crisis of Credit Visualized - HD https://www.youtube.com/watch?v=bx_LWm6_6tA&t=26s&ab_channel=graphixmdp
Investment Banking Will Survive The Crisis https://www.forbes.com/2008/09/15/goldman-morgan-banking-biz-wall-cx_pm_0915notes.html?sh=194f4bc02621
Owning the Consequences: Clinton and the Repeal of Glass-Steagall https://www.demos.org/blog/owning-consequences-clinton-and-repeal-glass-steagall
Bill Clinton on How Entrepreneurs Can Transform the Country： https://www.inc.com/magazine/201509/james-ledbetter/inc-interview-bill-clinton.html
Bill Clinton on deregulation: ‘The Republicans made me do it! https://archives.cjr.org/the_audit/bill_clinton_the_republicans_m.php
https://www.wsj.com/amp/articles/SB113392265577715881
What is the Yield Curve, and Why is it Flattening? https://www.youtube.com/watch?v=5L_zQGPNXOk
Relationship between bond prices and interest rates Khan Academy https://www.youtube.com/watch?v=I7FDx4DPapw&ab_channel=KhanAcademy
Introduction to bonds Stocks and bonds Khan Academy https://www.youtube.com/watch?v=Qh-M3_L4xYk&ab_channel=KhanAcademy
Japan: The Fading Economy https://www.youtube.com/watch?v=ErUQnd-YFGg&ab_channel=EconomicsExplained
Capital Requirements: https://www.investopedia.com/terms/c/capitalrequirement.asp
2020Dec IPO Mania Sweeps Over Robinhood Crowd and Stokes a 111% Rally。
Newyorker - Liberalism According to The Economist)
Andrew Carnegie Claimed to Support Unions, But Then Destroyed Them in His Steel Empire https://www.history.com/news/andrew-carnegie-unions-homestead-strike
Four Reasons Financial Intermediaries Fail https://www.youtube.com/watch?v=cdGA9t_PDpY&ab_channel=MarginalRevolutionUniversity
The Bank of England: https://www.bankofengland.co.uk/
Fed doesn t really need to buy corporate bonds https://www.bloomberg.com/opinion/articles/2020-06-15/fed-doesn-t-really-need-to-buy-corporate-bonds
Fed seems to skirt the law to buy corporate bonds https://www.bloomberg.com/opinion/articles/2020-06-18/fed-seems-to-skirt-the-law-to-buy-corporate-bonds
One percenters close to surpassing wealth of u s middle class https://www.bloomberg.com/news/articles/2019-11-09/one-percenters-close-to-surpassing-wealth-of-u-s-middle-class
One percenters close to surpassing wealth of u s middle class https://www.bloomberg.com/news/articles/2019-11-09/one-percenters-close-to-surpassing-wealth-of-u-s-middle-class
Milton friedman was wrong look at income inequality https://www.bloomberg.com/opinion/articles/2020-09-18/milton-friedman-was-wrong-look-at-income-inequality
Boj becomes biggest japan stock owner with 434 billion hoard https://www.bloomberg.com/news/articles/2020-12-06/boj-becomes-biggest-japan-stock-owner-with-434-billion-hoard
凯恩斯主义、货币主义和理性预期学派在失业和通胀关系问题上观点的主要差异 - 知乎 https://zhuanlan.zhihu.com/p/129648456
2008年金融危机的形成、泡沫和复苏全过程-虎嗅网 https://www.huxiu.com/article/281200.html
YOLO：不甘沉默的大多数_详细解读_最新资讯_热点事件_36氪 https://www.36kr.com/p/1075552592769921
Timeline: A short history of QE and the market - Marketwatch http://projects.marketwatch.com/short-history-of-qe-and-the-market-timeline/
一步到零，美联储为何如此激进？ - 专家观点 - 罗汉堂 https://www.luohanacademy.com/cn/insights/5e3da4ceb80664d6
一个简单的例子让你了解资产负债表！ - 知乎 https://zhuanlan.zhihu.com/p/118557325
如何看待主持人崔娃关于近期美警暴力执法引起骚乱等事件的发声？ - 知乎 https://www.zhihu.com/question/398493429/answer/1258461624
怎么理解输入性通货膨胀？ - 知乎 https://www.zhihu.com/question/277816021
如何看待亚当斯密的《道德情操论》和《国富论》？ - 知乎 https://www.zhihu.com/question/20879716
The 1870-1914 Gold Standard: The Most Perfect One Ever Created https://www.forbes.com/sites/nathanlewis/2013/01/03/the-1870-1914-gold-standard-the-most-perfect-one-ever-created/?sh=29ce5b374a6a
Are bank deposit liability or asset? - Quora https://www.quora.com/Are-bank-deposit-liability-or-asset
Modern Monetary Theory, explained - Vox https://www.vox.com/future-perfect/2019/4/16/18251646/modern-monetary-theory-new-moment-explained
504 GATEWAY_TIMEOUT https://advisors.vanguard.com/insights/article/whatsgoodforthefedmayalsobegoodforinvestors
Fed Reveals Which Bond ETFs It Purchased - ThinkAdvisor https://www.thinkadvisor.com/2020/06/01/fed-reveals-which-bond-etfs-it-purchased/
The financial crisis of 2008: How housing contributed - Curbed https://archive.curbed.com/2018/8/29/17788844/financial-crisis-2008-cause-housing-mortgage-lending
The Dark Side of Darwinism - Philosophy for the Many https://sites.williams.edu/engl-209-fall16/uncategorized/the-dark-side-of-darwinism/
Before ‘The Wealth Of Nations,’ Adam Smith Penned The Ultimate Guide To A Moral Life - HuffPost Life https://www.huffpost.com/entry/before-he-wrote-a-manifes_n_5772360
Reserve Requirements Definition https://www.investopedia.com/terms/r/requiredreserves.asp
Credit: How it is Created (Financial Economics) - Economics - tutor2u https://www.tutor2u.net/economics/reference/financial-economics-creating-credit
‘Wealth increase of 10 men during pandemic could buy vaccines for all’ - BBC News https://www.bbc.com/news/world-55793575
Accounting Equation Definition https://www.investopedia.com/terms/a/accounting-equation.asp
The Fed bought more blue-chip and junk bonds, and has started making Main Street loans https://www.cnbc.com/2020/08/10/the-fed-bought-more-blue-chip-and-junk-bonds-and-has-started-making-main-street-loans.html
Moneyness: The Idiot’s Guide to the Federal Reserve Interdistrict Settlement Account http://jpkoning.blogspot.com/2012/02/idiots-guide-to-federal-reserve.html
US Net Wealth - deconstructingrisk https://deconstructingrisk.com/tag/us-net-wealth/
Federal Funds and Interest on Reserves - Fedpoints - FEDERAL RESERVE BANK of NEW YORK https://www.newyorkfed.org/aboutthefed/fedpoint/fed15.html
Amazon Not Paying Federal Income Taxes on $11.2 Billion Profits - Fortune https://fortune.com/2019/02/14/amazon-doesnt-pay-federal-taxes-2019/
Amazon Just Killed a Tax That Helps Homeless People - Fortune https://fortune.com/2018/06/12/amazon-just-killed-a-tax-that-helps-homeless-people/
SPAC 是什麼？有什麼好處？運作流程與結構？SPAC IPO 上市為何在美國爆紅？
Thomas Jefferson and slavery - Wikipedia https://en.wikipedia.org/wiki/Thomas_Jefferson_and_slavery
Accounting Equation Definition https://www.investopedia.com/terms/a/accounting-equation.asp
FDIC: Insured or Not Insured? https://www.fdic.gov/deposit/covered/notinsured.html
Nonbank Financial Companies (NBFCs) Definition https://www.investopedia.com/terms/n/nbfcs.asp
If You Don’t Understand Banks, Don’t Write About Them https://www.forbes.com/sites/francescoppola/2019/09/17/if-you-dont-understand-banks-dont-write-about-them/?sh=3af754322e69
House Passes $3.8 Trillion in Tax Cuts - Fortune https://fortune.com/2018/09/28/house-3-8-trillion-tax-cut-passes/
How billionaires got $637 billion richer during the COVID-19 pandemic https://www.businessinsider.com/billionaires-net-worth-increases-coronavirus-pandemic-2020-7
The Dark Side of the Gilded Age - The Atlantic https://www.theatlantic.com/magazine/archive/2007/06/the-dark-side-of-the-gilded-age/306012/
How billionaires got $637 billion richer during the COVID-19 pandemic https://www.businessinsider.com/billionaires-net-worth-increases-coronavirus-pandemic-2020-7
乔治·佛洛伊德之死：从“我会改变世界”到“我无法呼吸”的一生 - BBC News 中文 https://www.bbc.com/zhongwen/simp/world-52988524
Could Blockchain Solve Our Growing Privacy Issue? https://www.forbes.com/sites/danielnewman/2019/05/08/could-blockchain-solve-our-growing-privacy-issue/?sh=3986b6d35eb4
The Rich Can’t Get Richer Forever, Can They? - The New Yorker https://www.newyorker.com/magazine/2019/09/02/the-rich-cant-get-richer-forever-can-they
Commentary: Amazon HQ2: How Jeff Bezos Avoids Paying Taxes - Fortune https://fortune.com/2018/11/21/amazon-hq2-tax-breaks-subsidies/
What are the fundamental accounting differences between capital requirements and reserve requirements set for a bank? - Quora https://www.quora.com/What-are-the-fundamental-accounting-differences-between-capital-requirements-and-reserve-requirements-set-for-a-bank
Understanding the Federal Reserve Balance Sheet https://www.investopedia.com/articles/economics/10/understanding-the-fed-balance-sheet.asp
What is a SPAC? - CNBC Explains - YouTube https://www.youtube.com/watch?v=jfNFI5JhrvA&ab_channel=CNBCInternational
The Rich Can’t Get Richer Forever, Can They? - The New Yorker https://www.newyorker.com/magazine/2019/09/02/the-rich-cant-get-richer-forever-can-they
Top 1% Of U.S. Households Hold 15 Times More Wealth Than Bottom 50% Combined https://www.forbes.com/sites/tommybeer/2020/10/08/top-1-of-us-households-hold-15-times-more-wealth-than-bottom-50-combined/?sh=53b2f0b75179
Milton Friedman On The Social Responsibility Of Business – Forbes Advisor https://www.forbes.com/advisor/investing/milton-friedman-social-responsibility-of-business/
Fact Sheet: Taxing Wealthy Americans - Americans For Tax Fairness https://americansfortaxfairness.org/tax-fairness-briefing-booklet/fact-sheet-taxing-wealthy-americans
Japan’s Banks Find It Hard to Lend Easy Money - WSJ https://www.wsj.com/articles/SB10001424052702304470504579163094082999108
How tax brackets actually work - YouTube https://www.youtube.com/watch?v=VJhsjUPDulw&ab_channel=Vox
Junk Bond Yields Hit Record Low: Most Distorted Markets Ever - Wolf Street https://wolfstreet.com/2020/12/04/nobodys-worried-about-nothin-junk-bond-yields-hit-record-low/
A better way to tax the rich - YouTube https://www.youtube.com/watch?v=pTwPHuE_HrU&ab_channel=Vox
How does US tax law allow billionaires not to pay? Read this and try to understand - CNNPolitics https://edition.cnn.com/2020/09/28/politics/tax-law-101/index.html
What is Equity - YouTube https://www.youtube.com/watch?v=Q1z395u60xU&ab_channel=TheFinanceStoryteller
Even as Americans Grew Richer, Inequality Persisted - The New York Times https://www.nytimes.com/2020/09/28/business/economy/coronavirus-pandemic-income-inequality.html
Here are 5 ways the super-rich manage to pay lower taxes https://www.cnbc.com/2019/02/21/here-are-5-ways-the-super-rich-manage-to-pay-lower-taxes.html
Herbert Spencer - American Experience - Official Site - PBS https://www.pbs.org/wgbh/americanexperience/features/carnegie-herbert-spencer/
各国都在争抢外国富人 - SWI swissinfo.ch
Amazon had to pay federal income taxes for the first time since 2016 https://www.cnbc.com/2020/02/04/amazon-had-to-pay-federal-income-taxes-for-the-first-time-since-2016.html
Society Has Become More Unequal Since Milton Friedman’s Day - The New York Times https://www.nytimes.com/2020/09/10/business/dealbook/milton-friedman-inequality.html
Repo: How Roughly $1 Trillion Moves Overnight - WSJ - YouTube https://www.youtube.com/watch?v=gzCkXNrjFQM&ab_channel=WallStreetJournal
Open Market Operations (OMO) Definition https://www.investopedia.com/terms/o/openmarketoperations.asp
How to tax the rich, explained - Vox https://www.vox.com/2019/3/19/18240377/estate-tax-wealth-tax-70-percent-warren-sanders-aoc
A Friedman doctrine‐- The Social Responsibility Of Business Is to Increase Its Profits - The New York Times https://www.nytimes.com/1970/09/13/archives/a-friedman-doctrine-the-social-responsibility-of-business-is-to.html
Quantitative Easing: How Well Does This Tool Work? - St. Louis Fed https://www.stlouisfed.org/publications/regional-economist/third-quarter-2017/quantitative-easing-how-well-does-this-tool-work
Chamath Palihapitiya explains why SPAC’s are awesome - YouTube https://www.youtube.com/watch?v=TO8WZgLTxkk&t=74s&ab_channel=InvestinYourself
Amazon in Its Prime: Doubles Profits, Pays $0 in Federal Income Taxes – ITEP https://itep.org/amazon-in-its-prime-doubles-profits-pays-0-in-federal-income-taxes/
40% of Americans don’t have $400 in the bank for emergency expenses: Federal Reserve - ABC News https://abcnews.go.com/US/10-americans-struggle-cover-400-emergency-expense-federal/story?id=63253846
FRB: H.4.1 Release– Factors Affecting Reserve Balances – Thursday, February 4, 2021 https://www.federalreserve.gov/releases/h41/current/h41.htm
The Rich Can’t Get Richer Forever, Can They? - The New Yorker https://www.newyorker.com/magazine/2019/09/02/the-rich-cant-get-richer-forever-can-they
Enclosing in God’s Name, Accumulating for Mankind: Money, Morality, and Accumulation in John Locke’s Theory of Property。
Corporate debt nears a record $10 trillion, and borrowing binge poses new risks - The Washington Post https://www.washingtonpost.com/business/economy/corporate-debt-nears-a-record-10-trillion-and-borrowing-binge-poses-new-risks/2019/11/29/1f86ba3e-114b-11ea-bf62-eadd5d11f559_story.html
Stock market ends year at record levels - The Washington Post https://www.washingtonpost.com/business/2020/12/31/stock-market-record-2020/
Jason Zweig WJS Columns: Jason Zweig - Investors Piled Into This Magical Money Machine. Now They’re Stuck.
Trump’s lawyer plays a video of Democrats and celebrities advocating violence at impeachment trial
郑戈-马基雅维利的三副面孔
尼尔·波兹曼 - 娱乐至死
The complete FUN TO IMAGINE with Richard Feynman - See new HD upload https://youtu.be/nYg6jzotiAc
The Roots Of Anti-Asian Racism
https://twitter.com/atrupar/status/1372203730481864713

“Yesterday was a really bad day for him and this is what he did” – a law enforcement official explains Robert Aaron Long’s decision to kill 8 people in a strange manner

2020年关于现代经济的一些思考(五)

2021-02-05T14:55:32+00:00

第二个原因是税收漏洞和对富人更友好的税率

我们以亚马逊Amazon为例子, 亚马逊一直有逃税的历史，在2017到2018年利润翻番从56亿到了112亿，而支付的税收是0.

民主党人桑德斯都忍不住吐槽。

特朗普2018年也说：

I have stated my concerns with Amazon long before the Election. Unlike others, they pay little or no taxes to state & local governments, use our Postal System as their Delivery Boy (causing tremendous loss to the U.S.), and are putting many thousands of retailers out of business!

实际上像亚马逊这种缴税那么少也是正因为特朗普2017上任后通过的3.8万亿大规模减税计划，不仅让公司税从35%减到了21%，而且没有关闭堵上那些避税的漏洞 -使得盈利丰厚的公司经常逃避联邦税同时申报的收入税只基于他们一般利润的一半。亚马逊还像上面提到了威胁地方政府西雅图停建在建的办公楼等等基础设施表示如果政府要征收帮助到流浪者的税，他们将会停止在西雅图的发展，转移到其他更友好的地区去。2019年亚马逊只交了1.62亿美元的税，只占它年收入的1.2%: Amazon Just Killed a Tax That Helps Homeless People.

不管亚马逊，从1980年到现在十亿富豪们缴纳的税降低了79%。除此之外，2017年估计有10%的全球GDP被藏在海外税收天堂。来看看美国的税率制度，是不是更有利于富人。2019： VOX - Who pays the lowest taxes in the US?解释了非常清晰，其中有一个截图：

税收里面有很多可以减免的，比如做慈善或者想办法亏损（比如特朗普2017年仅缴税750美元，他是如何做到的？）等等，其他的比如工资税，社保局缴纳的是有上限的，比如五险一金最多2000千。也就是我月薪四五千的人缴纳的比例可能远大于富人，而且消费税对于穷人的比例远高于富人购买奢侈品。富人缴纳的税比列相对来说反而比穷人还少 - 你认为是阶梯上涨的，其实总体而言更有利于富人。

另外巴菲特支付了比百万美国人都还要低的联邦税，因为投资赚的钱(unearned income)的联邦税是低于那些工资和薪水的联邦税。因为巴菲特赚的钱大比例都来自股票投资，所以他的收入税是比较低的，按照比列来说，甚至低于他的秘书和员工（工资收入），这话听起来像是胡诌的，但是这确是巴菲特本人2011年纽约时报发了一篇《请停止溺爱超级富豪Stop Coddling the Super-Rich》说的，他呼吁富余的美国人应该缴费更高的税。2019年的这篇文章里列举出了超级富人逃税或者减少税的五种办法:《Here are 5 ways the super-rich manage to pay lower taxes》，毫无疑问富人有更多的方式，精力和资源帮助他们减少税务的负担。

如何更好的向富人征税Vox - How to tax the rich, explained了？有议员Elizabeth Warren提议按照总资产的百分比来%2征收常年财富税（具体措施可以参考A better way to tax the rich听起来很合理）; Sanders桑德斯认为是可以升高房地产遗产税到77%；还有Alexandria Ocasio提出提高顶部富豪的税率70%等等；《21世纪资本论》的作者Thomas Piketty认为应该大幅度加税包括遗产税，税收应该是用来最大化公众的福利，所以你应该只从富人那里收税来推动社会的福利，甚至建一个全球联合起来统一制定向富人征税的合作机制，防止从一国跑到另外一国（比如这个作者的母国法国，奥朗德时代部分富人税一度高达75%，导致法国富豪纷纷跑到税率优惠的邻国瑞士） - 不过有点理想化了。听起来很合理，但是现实不太可能，正如某个评论里说的：即使美国政府有从富人加征税拿到了这么多钱，可能也不会用到上面提到的教育公共福利减少贫困，可能更多的还是用到军费上面。

2021年Aug 5的文章《Wealthy people are renouncing American citizenship》提到为什么富人放弃美国国籍：

世界上只有美国和厄立特里亚两个国家，是根据公民身份而不是居住地征税。只要你是美国人，不管住在哪里，都必须向美国政府交税。其它国家的规定是，本国公民如果住在其它国家，则无需向本国政府交税。

这导致大量的美国富豪放弃美国国籍，申请其它国家的公民。谷歌公司的创始人拉里佩奇就申请成为新西兰公民，前 CEO 埃里克·施密特（Eric Schmidt）申请成为塞浦路斯公民。

疑问

1.为什么印了这么多钱，但是没有通货膨胀？

上面也提到了弗里德曼也说了通胀是因为钱太多了。为什么美联储从08年到今年的QE印了几万亿的钱，投放到市场中，怎么就没看到通货膨胀率飙升了，甚至平均起来都打不到标准的2%？日本这么多年QE（甚至都已经直接购买了股票ETF，到2020年12月日本的中央银行BOJ已经成为日本股市最大的持有者，高达4千亿美元BOJ Becomes Biggest Japan Stock Owner With $434 Billion Hoard，自由市场好像不太自由了，怎么混进了奇怪的东西）上文中也提到了美联储的研究发现日本的通胀率也没有上来其实还是通缩，这是为何了？

按照央行的思路，理想情况下，低利率->廉价资金->借贷成本低->公司借钱扩展投资->更多产品输出->经济增增日上->通货膨胀。

但是现实是，央行本质是金融的媒介，媒介的传播是有速度的，有的顺畅有的堵塞；好比一个人给他的饭是够了，但是他食道或者胃就那么大，只能消化那么点，你给他更多饭食物，他也没办法很快吸收消化。

美联储宽松的货币政策，表现在基础货币的大幅度增加，并没有带来广义货币的大幅度增加，传导到通胀率更是有限，还达不到美联储的通胀目标。因为真正决定货币供给的，不但包括基础货币的体量，还包括货币的周转频率。当消费者和投资预期悲观的时候，市场不会买东西和投资，那么即便基础货币很高，周转速度会大幅度下降，从而抵消基础货币量增长的效果，并不会带来真正广义货币的增加，也不见得会转化为成通胀。量化宽松和通胀并没有必然的联系，从理论到实际都是如此。

如果量化宽松不会带来货币的大幅度增加，那么是不是就不需要量化宽松了呢？在美联储看来，如果不增加基础货币和流动性，再加上货币周转的大幅度下降，可能带来货币供给的灾难性下降。这是美国在20世纪大萧条时期学到的教训。

先看看基本的经济学基础经典的费雪方程式来理解。费雪方程式即MV=PT，M是货币供应量，V是货币流通速度，P是平均价格水平(可以衡量通胀水平)，T是商品和服务交易数量，为了方便说明，价格水平可以表示成P=MV/T。

假设V也就是流通速度不变的情况下，M货币供应量的提升确实可以带来价格水平P的升高（更别说危机的一段时间代表T的商品和服务交易数量萎缩）；但是在显示经济形势了V可能会跟着变化，比如危机中，有效需求降低，债务的压力进一步消费欲望骤然下降（想想底层人民 - 有些人光是活着就已经拼尽了全力-这些人是大部分，不是B站说的那种光鲜亮丽的后浪-少数），那么货币流转流转的速度会大幅度下降-结合银行存在美联储储备银行中多余储备金的数量，这样一来就不一定带来通货膨胀。看看日本吧，日本的银行甚至很难找到人去放贷，尽管贷款利息很便宜，你不会因为利息便宜就去借钱对吧：《WSJ - Japan’s Banks Find It Hard to Lend Easy Money》。在这里真心建议日本央行学习下中国这些金融放贷广告的套路：《京东金融广告 360借条广告土味脑残金融广告合集远离网贷，拒绝鼓吹消费升级！》.

美联储设置2%的通胀率目标，而这十几年通胀率几乎都是低于2%，甚至有点距离，其目的是让人们担心未来会发生通货膨胀，因为通胀的预期赶紧花出去自己手中的钱（假设0通胀的债券利率是4%，现在未来通胀会有5%，那么投资者肯定会预期长期利率9%，所以现在借钱花钱划得来），理论上这会引起通货膨胀，实际情况看起来并不是这样，更有点像凯恩斯说的流动性陷阱(liquid trap).特别是当你短期利率为0的时候，意味着现金和债券几乎没有区别，成为了实际上相等价的资产，所以传统的货币政策改变-即通过公开市场交换钱和债券的操作-改变不了任何事情。这就是保罗克努格曼Paul Krugman在1999年《THINKING ABOUT THE LIQUIDITY TRAP》提到的：

“If the interest rate is zero, bonds and money become in effect equivalent assets, so conventional monetary policy, in which money is swapped for bonds via an open-market operation, changes nothing.”

使用纯货币政策来解决社会总需求的不足就像凯恩斯说的: 买一根大的皮带来尝试给你增肥like trying to get fat by buying a larger belt。纯粹的将钱从美联储转移到社会手中成为存款在经济上是绵软无力的，没有提高的需求的存进，钱没有流动起来，就像里根的减税政策的效果比共和党人想的要差，实际更多是效果来自里根的军事花费的增长。

当然也不能排除像弗里德曼说的通胀可能像是年轻人在聚会上喝酒，一开始（政府花钱花的很爽）喝的很爽酒带来的赶紧很high嗨，但是真正难受的是第二条早上的宿醉，那种头晕晕沉沉特别难受（通货膨胀）- long and variable lags用他的话说需要6到16个月的时间货币政策的实际效果才会显示。

用前美联储主席海伦Janet Yellen耶伦的话说Yellen sides with Milton Friedman against colleagues：

Yellen did acknowledge that the workings of the economy may have changed at some point since Friedman’s statement. “The structure of the economy changes, things do change,” she said, noting as an example that “we’ve seen inflation respond less to the economy, to movements in the unemployment rate.”

所以有可能QE就像喝了很大很强的洋酒，酒劲可能隐藏十年都不一定能出来，但是保不准一旦出来那必定是惊天动地，或者通胀可能已经潜移默化的变形为其他的方式，不是他们传统盯着的衡量标准。08年的财政支出和货币政策的扩展QE都没有让通胀在08-20中间得到他们预想的升高（当初保守主义者猛烈批评这会导致巨大的通货膨胀，但是出乎意料一直没有出现），到2020年两只铁拳又一次联手更大规模更大力度的出击，但是截至目前看来更像是通缩。

这中间一定是出了什么问题。

2.钱怎么来的？

最后我们要回到正题 - 钱怎么来的。

回想开头一下在1700年代的英国人民把黄金存到金铺然后金铺给他们一张IOU我欠你xxx克黄金凭此票见票即付，然后后续客户可以凭借词条见票即付，从金铺里拿回自己的金条。在美国1800年代没有中央政府之前有一阵子时间都是地方银行各自发行自己的钞票，当然钞票理应当跟他们自己金库里的黄金能对应上，有的储户于是将自己手头的黄金存到银行，可以获得存钱的利息，而银行-作为主要是赚取利息差的中介，他吸收个人用户存款在放贷给比如需要投资建厂的企业工厂等等，收取放贷的利息（通常放贷的利息高于存款的利息，更长期的贷款利息也越高），但是他也不能无限量发行钞票的放贷。事实上当经济情况不错的情况下，银行就跟陷入困难时期的政府一样，容易超发货币，进行更高收益率更高风险的投资。一旦比如社会上出现季节性用钱，比如美国当时主要的农民在特定的秋天丰收季节或者春耕需要从银行取存钱的时候，就容易造成银行有时候无法将长期的投资短时间兑换为黄金或者因为短期抛售资产价格下降等等，另外一个是银行因为追求利润可能将所有存款都放出去贷款了，没有多少存量，导致民众恐慌，谁都不想自己的存款被别人先取出来，而自己去的时候成为击鼓传花的最后一棒，进而挤兑银行，银行因为没有足够的黄金导致破产。后来美联储成立，规定银行放贷的钱不能超过吸收的存款的10%，也就是银行吸收存款比如$100美元，只能放贷出去$90, 必须保留$10来满足普通情况的部分储户取款的需求。

1920年年代的飞速发展使得银行放贷的越来越多，信用宽松，甚至有的低于了准备金的比列，当大萧条发生时，所有人差不多一起取钱，那么这个比列无法满足特殊情况了，人心惶惶人们都跑去银行挤兑，这个时候这些银行理论上应该被美联储注资救助（Lender of Last Resort)，但是反而并没有，因为大部分银行要么地区性的私人的要么不在美联储的银行体系里，要么没有足够符合美联储接受条件的抵押物，从1930到1933年超过一千家银行倒闭，这些银行的倒闭进一步恶化了经济形势。知道1933年三月，上面提到的罗斯福颁布的银行法，全国放了银行假日，将银行都纳入到政府的体系里（政府授权），同时政府建立一个临时的储蓄保险系统 Federal Deposit Insurance Corporation (FDIC) 后面变成了一个长久的系统（正式因为此，在1980s年代末期因为金融管制的松懈，存款机构投资于高风险的投机引发的危机并没有造成恐慌1989 - Savings and Loan Crisis）。

看看脱离金本位之后，到现在有什么变化了。很明显到目前2021年，很少人在用纸币了，可能在国内感觉明显一些，微信和支付等等扫码支付基本电子化是主流了，在国外可能纸币用的频率还是要稍微高点了，尽管他们比我们更早广泛使用信用卡。所以总共有多少比列用现金，多少用数字了？没有精确答案，大概的话是92%用数字，其他的8%是物理实体的比如纸币。数字货币相对纸币有什么优点了？方便携带（不容易凡尔赛），干净卫生（抠脚的），结算方便（运钞车运来运去还容易被抢）和更大的透明性（比如贪官藏在床底或者墙壁里）。还记得我们说美联储货币政策主要常规工具，调整基准利率 - 在公开市场购买和卖出政府债券从而降低短期利率，这个过程还是比较模糊的，那么它究竟是怎么工作的？电子货币在里面扮演了什么角色了？

回忆一下，美联储董事会下面有十二个地区的美联储储备银行，每个储备银行管一个地区，每个地区里的银行都可以在储备银行存一定数量的储备金(Reserve) - 可以简单解释这就相当于投名状，你上梁山需要有杀一个人作为见面礼，不然我们好汉不认你。举例来说，美联储在公开市场从银行那里买入政府债券，将储备金(Reserve) 释放给银行，这里银行是无法直接消费储备金(Reserve) ，因为这个只是梁山上的内部协议，无法直接在比如二级市场上卖给或者买给个人投资者或者非银行的金融机构或公司Nonbank Financial Companies (NBFCs)（比如投资银行，养老基金，保险公司，抵押贷款机构，对冲基金，p2p放贷人等等），这个媒介只限于梁山上。简单回忆一下：

这里还需要了解下什么是资产负债表balance sheet. 关于这块可以看看这篇文章入门《一个简单的例子让你了解资产负债表！》.简单的说包括资产Assets（拥有的资产房子车子或者后续别人会给你钱的-借条啊等等），负债Liability（比如你像银行借的钱，房贷，车贷你会付钱出去的），最后是权益Equity（包括股东权益，挣得利润比如放贷获得的利息）。

下面是一张比较典型的图表，可以看出资产=负债+权益。如果这个人资产多，负债少，那么对应的权益也多，这个是比较理想的；相反那么就难搞了，债务多，资产少，入不敷出，恐怕难以为继哦，这是我们从普通和公司的角度的思考的。比如有你一个资产是房子值300w,然后你自己有储蓄大概100W，向银行借了200W放贷，这个时候资产是300W房子，债务是200W放贷；这个时候权益也就是300-200=100W；这个时候突然房价下跌，只值50W，但是你的债务是不会跟着资产价格下降的，这时候你的权益就只有50W-200W=-150W甚至是负的，那肯定很亏，那你想还不让银行没收房子得了，我不付贷款了，反而少亏50W；相反房子价格上升，比如到了1000W，你的权益就是800W，那肯定乐意；

从美联储的资产负载表可以看出来：

上面是资产主要包括政府债券和住房抵押债券（这些债券都会打来利息收入，到期可以兑现）；负债主要是储备金(Reserve) 和在流通中的货币Currency-比如你我持有的人民币或者美元；

有兴趣的可以看看FED官方1月28号的资产负债表，里面有非常详细的信息。这里要解释下为什么储备金(Reserve) 是美联储的负债了？上文讲到美联储公开市场从银行购买政府债券，那什么给到银行了就是储备金Reserve(储备金不是流通中的货币比如人民币美元)，他不能直接参与流通，像上面说的是投名状，带利息的压金或者说是银行存在美联储分行的存款，会有利息的。储备金Reserve可以说是美联储给银行打的收据IOU，银行只能存在各个地区的美联储储备银行（当然各个区域也会清算），美联储付给一定利息给银行。所以储备金(Reserve) 是负债，也就是说对于银行方面而言，存款是负债（需要未来不断付利息的），而不是对于个人或者存款方而言认为的资产。

当美联储去银行购买1000债券前，银行的资产是债券；购买时美联储从银行那里结果资产变成了自己的资产债券+1000,然后在负债栏上加一行储备金-1000（就相当于银行存1000在央行这里）；这时候银行的资产上+1000储备金；每当央行资产收到债券的利息时，它后面也需要在负债里给对应的银行打去（银行的存款）储备金的利息；当然一般来说政府国债的利息没有准备金的利息高的。所以银行不直接在自己保险库像过去持有黄金一样持有储备金，都是放在地区区域所在的美联储储备银行里。相反的，当美联储现在要卖出债券到银行，那么银行资产上划掉储备金1000，然后增加一条债券；美联储这边资产上划掉债券，同时在负债中将对应的的存款记录-1000划掉。这里讲的划掉是因为目前几乎都是数字化的，所以几乎就是敲几下键盘。

OK银行拿到了这些只能在梁山内部转让的储备金，在下面那套系统里，它如何影响现实当中的流动的货币了？

假设你现在去银行借钱买房子，一般流程是你需要提供比如公司开具的收入证明和去银行拉近六个月的流水，银行审核你有没有足够的能力负担得起你要借贷的钱，然后签署一系列代表法律效应的的文件和声明（比如如果每月放贷延期了多久会如何，什么情况会没收房子等等），等所有条件通过了，银行就会把钱打到你的银行账号上，通常是同一个银行的（如果没有账户，就需要开一个新户），过阵子就可以收到一个短信说钱到账了。随后你就可以去ATM取出现钞或者转账到卖主的手中或银行账号上。看看下图银行的资产负债表：

Tuo去银行借了1000块，这个时候银行会在资产表写入一条贷款(给tuo贷款1000)记录（贷款load对于银行而言是资产，因为借贷人后续需要陆续支付利息和本金），接着会在负债liability右边栏写入一条存钱(往tuo的账号存1000)的记录（deposit存钱对于银行是债务，因为银行需要打钱出去给借贷人）。稍等，但是，这个好像有点问题吧? 难道不是银行拿其他的存款拨付给借款人么？

这个就是违反常识和直觉的地方。商业银行在放贷的时候，不是从其他储蓄的存款里取钱（已经是债务了liability)，而是相等的在资产loan对应的加一条新存款的负债条目。这个”钱“不就是无中生有出来的么？银行在确认你签字的时候，就在他的电脑按几个键输入这些数目的记录就好了。凭空创造出钱？这个也太奇怪了，有什么根据了。

正如跟其他人一样太好奇了，做这个调查时候我也查了不少资料，有不少阴毛论的，也有比如那种Crackpottery-众人皆醉我独醒的逆向思维者(动机不清楚了)等等吧，还有一些来源不明的，这里我只根据比较权威的英格兰银行BankofEngland官方的资料Quarterly Bulletin 2014 Q1- Money creation in the modern economy有兴趣的可以看看（我这里没找到美联储的，下面的引用都是引用自这篇文章）。引用其中的关于商业银行放贷时：

Whenever a bank makes a loan, it simultaneously creates a matching deposit in the borrower’s bank account, thereby creating new money.

银行里面的储备金Reserves只能是银行之间的流通不会跟客户有直接接触。现代经济里，主要的存款来源不是储户存进银行的存款，而主要是银行房贷时创造的存款。储备金和放贷loan有什么关系了？这两个体系如何联动了？

银行首先决定放贷的多少取决于有多少可以获利的借钱机会，这其中就有英格兰央行设定的利率，决定了借贷成本和利息。类比一下美联储设置基准利率，但是一般来说美联储是先声明他希望的基准利率是多少，而后过一段时间才会具体召集银行操作，一般而言银行会在美联储声明期望值的时候就主动将利率调整到目标值（银行都知道who is your boss），所以一般不用等到具体召集开会，利率调整就已经到位了。回忆一下基准利率Fed Fund Rate:

The federal funds rate refers to the interest rate that banks charge other banks for lending to them excess cash from their reserve balances on an overnight basis. By law, banks must maintain a reserve equal to a certain percentage of their deposits in an account at a Federal Reserve bank. The amount of money a bank must keep in its Fed account is known as a reserve requirement and is based on a percentage of the bank’s total deposits.

Like the federal discount rate, the federal funds rate is used to control the supply of available funds and hence, inflation and other interest rates. Raising the rate makes it more expensive to borrow. That lowers the supply of available money, which increases the short-term interest rates and helps keep inflation in check. Lowering the rate has the opposite effect, bringing short-term interest rates down.

这个美联储设定的利率会影响到商业银行决策贷款与否，进而影响到创造出来的银行存款。反过来这些银行存款会影响央行决定银行该持有多少的储备金Reserve(来应对比如公众储户取款，支付给其他银行或者应对监管方面的流动性要求等等)。

在综合起来看，广义货币（Broad money)主要由银行存款bank deposit(实际上是商业银行发行的给公司和家庭的IOU欠条借据我欠你xxx - 将来你来我给你还给你) 和流通的现钞钞票currency(这个人民币banknote是央行发布的，也就是央行发行给社会大众的IOU欠条)。在两者中银行存款占了97%，这些银行存款主要是由上面描述的商业银行创造的。

b站, youtube

我们看到当用户借贷而给银行创造存款的时候，对于银行而言就是负债慢慢变大，为了应对比如储户取款或者支付给银行的需要，银行需要持有更多央行的储备金Rerserve才行。这时候储备金就是有央行按需要供给的，银行出售给央行其自己的资产比如国债等等来换取储备金。储备金数量不直接限制银行放贷和创造银行存款的数量。

This description of money creation contrasts with the notion that banks can only lend out pre-existing money, outlined in the previous section. Bank deposits are simply a record of how much the bank itself owes its customers. So they are a liability of the bank, not an asset that could be lent out. A related misconception is that banks can lend out their reserves. Reserves can only be lent between banks, since consumers do not have access to reserves accounts at the Bank of England.

当储户还贷时，这些时候银行会把该用户的储蓄账号的钱减少，这个时候新创立的这些钱就被销毁了（一笔勾销）。当储户大规模取钞票currency时，银行可能还需要向央行卖出自己的资产（主要是国债）来获取比如currency钞票banknote等给到个人。当银行向非银行的私人行业购买国债时，他们给这么卖出人的存储账号填写对应的数字就可以了（银行往卖出人户头存钱），当央行也进行类似的资产购买时，这就是QE了（绕过银行）。

上面的例子中比如我拿到了借贷的钱我如果直接在线转账转给卖主的银行会发生什么了？最简单直观的方式就是买主银行把自己在美联储储备银行里的储备金按照数目给卖主银行，然后卖主银行在卖主的储蓄账号上输入对应的数字。这样一来，当银行之家定期审查的时候，有的银行可能储备金高，有的银行可能少（比如河北的钱都跑到了北京去了，虹吸），这个时候银行可以向其他银行借钱（还有通过repo)，来符合监管或者储备金的最低比列要求。这个储备金基础要求(Reserve Requirement)在英国加拿大香港澳大利亚是没有的，也就是0，但是他们有资本充足率capital requirement要求（美国自从08年金融之后的监管改革Dodd-Frank也要求美国银行需要有基于风险模型评估的资本充足率要求）；在美国有，但是到了2020年3月在新冠病毒的危机下美联储已经将其降为0，也就是没有最低储备数目的要求的了。

所谓资本充足率capital adequacy requirement简单的说就是银行做的贷款是有风险的，根据贷款风险评级给风险因子，比如国债风险很低就是0，比如房贷的风险就比较高（这里面政府支持的GSE的房利美风险因子就小点0.2，其他的抵押贷款机构就是0.5），然后结合贷款的金额做一系列计算（实际比如Basel巴塞尔协议是非常复杂），根据风险资产的总值按照的一定比例(资本充足率）需要银行有足够的资本缓冲。当资本充足率越高，那么银行越能抵御风险和危机（absorbing loss);反正则越差，这个也是所谓的Stress Test压力测试；相反当银行进行越多的高风险投资和持有高风险资产时，那么资产充足率就会下降，如果低于监管的要求，那么就需要减少风险资产的持有，从股东那么融资甚至面临倒闭。

Under the Dodd-Frank rules, each bank is required to have a total risk-based capital ratio of 8% and a tier 1 risk-based capital ratio of 4.5%. A bank is considered “well-capitalized” if it has a tier 1 ratio of 8% or greater and a total risk-based capital ratio of at least 10%, and a tier 1 leverage ratio of at least 5%.

问题来了，理论上商业银行可以凭空随意在自己的资产负债表里捏造数据条目，是不是在现实中商业银行可以没有限制通过借贷的创造钱了？

不能。特别是借贷的利息 - 由人们借钱而决定。银行创造钱是有限制的，第一个是银行本身面临他们能借多少的限制：比如因为每个独立的银行都必须要通过自由市场竞争来是的借贷利率保存一个合理共识的水平来获取利润（你不能无限制的出高价，否则就被邻居抢走了），再者银行本身放贷本身也有风险（比如贷款人还不起钱破产或者资产价格大幅度下降缩水，这些因素都会被考量进入利息的计算里），监管也会限制他们确保他们不能无限制的放贷-比如都是高风险高回报的放贷但是资本缓冲的去很少；第二个创造的钱使用权在借贷人手上，他可以立马还贷来销毁刚刚新建的钱也可以花费消费也就是乘数效应造成经济通胀的压力；第三个最重要的还是央行的货币政策来影响短期利率，通过利率传统进而影响社会其他领域的利率。

Central banks do not typically choose a quantity of reserves to bring about the desired short-term interestrate.(4) Rather,theyfocusonprices—setting interestrates

TheBankof England controls interest rates by supplying and remunerating reserves at its chosen policy rate. The supply of both reserves and currency (which together make up base money) is determined by banks’ demand for reserves both for the settlement of payments and to meet demand for currency from their customers — demand that the central bank typically accommodates

这是理论正常情况上的，但是实际中比如监管，资本充足率的要求，当经济上行时，银行实际放出更多贷款，也获得更多的利润用来缓冲，使得资本充足率其实是也可能同步升高，同时这个风险评估也带有一定的主观性，特别是比如08年那种金融衍生已经玩的非常复杂，以至于没人可以计算出它的风险该是多少。总的来说，银行还是有了更多的限制，降低了风险。

Changesininterbankinterestratesthen feed through to a wider range of interest rates in different markets and at different maturities, including the interest rates that banks charge borrowers for loans and offer savers fordeposits.

一般情况下：央行降息 -> 银行借钱的利息下降 ->消费者一看借钱便宜 -> 借钱买车买房->贷款变多 ->银行之间结算需要更多的储备Reserve -> 卖出自己的资产从央行得到Reserve -> 结果Reserve变多。

而不是一般意义上认为的央行直接主动放水，应该来说是间接的，主要目标和出发点不是货币数量而是利率 - 其实也就是通货膨胀率。所以美联储说它的目标是维持合理的通胀率2%，而没有说以买多少国债为目标。

b站, youtube

这是完整的从买方贷款到卖方收到钱的完整交易过程，出来上面的个人负债表还有下面对应的其所在银行的负债表变化情况。

但是当短期利率几乎降为0时，这时候的货币政策就没有是么作用了，因为利息降无可降，至少美联储是这么认为负利率是不可能接受的，这样一来按照上面我描述的那个逻辑链条，将不会发生更多的借贷和消费。这个时候就需要跳过这些银行了，就好比原来是央行->银行->私人，现在银行这中间这段堵住了，又没办法绕过去了直接输入到私人领域了，来进一步刺激经济了。

b站, youtube

QE这个时候央行的目标就是钱的数量了。上面提到非银行的金融机构或公司Nonbank Financial Companies (NBFCs)（比如投资银行，养老基金，保险公司，抵押贷款机构，对冲基金，p2p放贷人等等）持有国债，这里假设是养老基金持有国债，简单的说，它可以卖给央行，央行直接在负债栏目里创建储备（Reserve)，同时将国债列入资产，这样可以把储备（Reserve)以存款的方式发给养老基金。而养老基金可以拿到这些钱去投资更有风险更高回报的领域，比如购买公司债券或者股份等等，从而推高资产价格同时降低企业融资的成本，反过头来应该可以导致更多经济的消费。

QE会直接影响广义货币（商业银行）和基础货币(央行)。但是养老金和其他私人非银行机构一样它不是美联储的成员银行，没有储备金账号，他不能直接跟央行兑换储备(Rserve），只能把商业银行拉进来作为中介辅助完成交易，也就是中间有商业银行但是这个角色在这中间只是一个工具人。

上面显示养老金将持有的国债变成了在商业银行的存款，负债不变；央行为了购买国债创建了负债储备(Reserve），而资产有了国债；中间商商业银行则持有央行的创立的储备金作为资产（IOU from BoE)，负债则是养老金的存款；

关于QE的误区一个观点是央行给银行“免费的钱”（QE involves giving banks ‘free money’）。这个其实不对的，虽然央行给了储备金给商业银行，但是商业银行持有了负债就是养老金的存款，央行要支付给银行多余的储备金的利息，但是这部分利息银行也需要转移给养老金用来支付存款的利息，所以这里面商业银行只是一个单纯的媒介intermediary中间人来方便完成交易；

第二个误区是QE主要目标是通过给银行系统提供更多的储备金Reserve来增加银行的借贷。这个也很清楚，商业银行作为中间人是增加了储备金Reserve，但是商业银行是不能将储备金直接借给消费者的（没有reserve account账号的)，储备金是梁山上的规则，不能适应于朝廷。当银行放贷时候，存款对应数量的增加，但是储备金数目是不变的。

总的来说QE是绕过银行行业，针对的是私人行业消费水平的提高。这里讲到的还是国债，上面之前讲到的，其实美联储已经直接向个体公司和持有公司债券的ETF等直接购买债券，意在直接把钱丢在你的手里 - “我是你爸爸，拿着去消费的把“。更有甚至日本央行已经直接购买公司的股票，而不是债券了，相当于间接成为了是公司的股东，当然我相信日本央行肯定不会介入啥公司日常运作或者投票表决咯。

上面是现代经济中钱是如何产生的。商业银行凭空(ex nihilo)产生钱这句话是对的，但是正如米德尔曼说的没有任何事情是免费的There’s No Such Thing as a Free Lunch，这个钱要么是原地还回去销毁或者进入市场流通跟其他银行交换就会动用银行之间在央行的储备金Reserve进行清算 - 理论上它们需要在央行持有同等的储备金Reserve。即使英国央行(之前都会有voluntary reserve target自愿性质的储备金目标）和2020年之后的美联储都取消了最低储备金的要求(Reserve Requirement)，也就是0；有意思的是08年之后反而他们在央行持有了大量的储备金，甚至是过多过量；再一个即使没有最低准备金要求，也会有上面讲的流动性要求 liquidity requirements就是巴塞尔协议的资本充足率-来覆盖至少30天内的取钱和其他的支付。

这么来看Banks create money out of thin air.银行凭空生钱听起来骇人听闻，而且特别容易跟阴谋论联系在一起，但是经过上面的分析感觉好像这句话只是交代了事情的一部分而已，如果我们捋直了来看其实还是没什么毛病的。尽管单个人来看，我们个人是弱者，银行是强势者，只是看问题不能只是“我弱我有理，你强肯定是你黑”，当然不排除有其他不可告人的因素，但是理性的来说我们还是根据已有的资料和证据结合现实的观察来分析推导得出最符合逻辑的结论（当然如果你说我得来的资料都是英国央行的，都是不可靠的，所以你不用白费心思做逻辑研究了 - 那我无话可说-good for you）。举个例子，比如网上很多讲到银行和货币系统的黑暗规则时都会提到汽车大王亨利福特Henry Ford的一句话(比如在油管上有个ColdFusion的博主里发布了一篇《Who Controls All of Our Money?》就一开头引用了）：

“It is well enough that people of the nation do not understand our banking and monetary system, for if they did, I believe there would be a revolution before tomorrow morning.”

这句话煽动力真是max.其实这句话的来源就有问题的，没有指向和明确的资料可以说明是福特说的： Did Henry Ford say this quote, and what did he mean by it?_（一个1957年说是据xxx说，另外一个1959年出现是在一本美联储阴谋论的书里）- 事实上福特的传记等等都没有这话。

总结

美联储当然可以这么做将钱扔到公司企业的手中，一方面让他们保住就业，别裁人，一方面希望公司扩大生产消费，让经济好起来。但是当经济下行之中时，社会的需要不振，消费欲望下降，这些不是那些公司高管或者超级富豪可以买点奢侈品游艇就能提振的，说到底还是普通人，那群40%都在银行卡里拿不出400现金救急的底层人物和中产阶级，毕竟富豪总共人才有多少。看起来富豪更喜欢将钱投入到股市回购或持有更多的股票基金等资产来获取低风险稳定的高回报（想想吧这本来应该是高风险的），而不是投入到寄希望于前途暗淡的消费市场复苏而雇佣更多人更多厂房生产更多的商品。很多人认为美联储在08年bailout那些作恶的大银行，在2020年要放水让那些富豪的财富以惊人的财富上涨，并没有普惠到更多底层的人。上面提到的连鲍威尔在2021年1月27号的发布会都说了特别关注那么低收入，非洲裔美国人和西班牙裔的就业情况。

正如查理芒格所说的：

对于拿着锤子的人来讲，全世界都是钉子。

对于经济学家而言（不一定是经济学家，普通人也是）当一门理论或者实践一开始能帮助打破当前的迷局时，他们可能会迷信这门理论可能在后面所有的迷局都可以适用，抱着过去这个可以工作现在也应该可以的态度去过渡的使用，结果发现貌似这门理论或者方法好像不太好用了 - 这个时候所有人都要在找另外一种方法。好比凯恩斯主义比较适用于短期的经济危机，好比强心针，但是不能当做长期维生素片天天吃，使用的越长副作用越大，但是架不住危机二战时候效果太好了，从这个角度叫滞涨的发生也是必然的。货币主义从被嘲笑到逐渐登上舞台成为主流，弗里德曼货币主义的背后是新自由主义-捍卫资本主义：放任（laisser faire ）自由市场，强调个人主义individual，政府应该放开管制尽量少插手。自由市场的竞争才能确保资源的最佳分配和效率最大化，认为政府才是形成垄断的罪魁祸首。

他喜欢用的一句话: Nobody spends somebody else’s money as carefully as he spends his own.所以他认为政府应该是最低限度程度，是一个中介尽此而已。所以里根就职典礼上致辞才会说： “Gov is the problem”，撒切尔才会说：”根本就不存在社会这种东西there is no such thing as society” 。政府里的福利政策让人失去了效率，太多的free rider搭便车不干事的，只有让他们没有了政府提供的安全网(safty net)，让他们没有了保障意识到靠自己才会努力竞争发挥出最大的激发他们的能力，淘汰那些不符合市场的落后的不思进取的个人，反而能达到更高程度的公平。

1987年撒切尔在《Interview for Woman’s Own 》采访时说道：

“I think we have been through a period when too many people have been given to understand that when they have a problem it is government’s job to cope with it. ‘I have a problem, I’ll get a grant. I’m homeless, the government must house me.’ They are casting their problems on society. And, you know, there is no such thing as society. There are individual men and women and there are families. And no governments can do anything except through people, and people must look to themselves first. It is our duty to look after ourselves and then, also, to look after our neighbours. People have got their entitlements too much in mind, without the obligations. There is no such thing as an entitlement, unless someone has first met an obligation.”…

That was the objective, but somehow there are some people who have been manipulating the system and so some of those help and benefits that were meant to say to people: “All right, if you cannot get a job, you shall have a basic standard of living!” but when people come and say: “But what is the point of working? I can get as much on the dole!” You say: “Look! It is not from the dole. It is your neighbour who is supplying it and if you can earn your own living then really you have a duty to do it and you will feel very much better!”

联系采访上下文，撒切尔的意思是贪婪没有问题，问题是政府（父母）的很多福利措施溺爱了个人（孩子），是的他干什么事情第一个都是找政府求助，而不是靠自己本身。比如有的人说干嘛去工作，我有政府提供的失业保险津贴啊，但是这个津贴是来自己社会上其他人比如你的邻居贡献的；如果你真的可以靠自己生活，那么你有责任去这么做而且你会感觉好很多。

总的来说新自由主义Neoliberism强调小政府，放松监管，自由市场竞争。新自由主义来自古典自由主义Liberism,起自John Locke洛克，Adam Smith亚当史密斯和密尔(John Stuart Mill). 相对当时的重商主义，新自由筑强调自由贸易，支持资本主义。鼻祖John Locke洛克在主张有限政府和私人财产权和信仰自由同时，却尝试正名化奴隶贩卖，拒绝印第安人的财产权来方便掠夺土地(ps:这块查证了下感觉有争议)；亚当史密斯因为国富论的无形的手 - 自由交换和劳动分工-被称为资本主义之父，但是其更早另外一本书道德情操路则被惨遭丢弃；密尔曾在东印度公司工作支持帝国自由主义，认为印度人太原始太野蛮了(native character . . . half child, half savage, actuated by sudden and unreasoning impulses.)不太配享有自由权利，应该有优越的白种人-英国人-来教印度人如何治理印度；1843年的《经济学人》(The Economist)的创始人詹姆士·威尔逊(James Wilson)自由主义的拥护者（此本杂志的立场）他本人拒绝禁止奴隶贩运理由是这样会影响英国消费者，1840年代的天主教的爱尔兰遭受饥荒，英国坚持自由贸易出口爱尔兰的食品-威尔逊认为需要更多顺势治疗(homeopathic remedy)-自由贸易，他坚持英国强力镇压爱尔兰的暴乱；同时他忽视英国国内的不平等，认为政府责成铁路公司去改善工人乘客的服务，使得工人们不用受苦地坐着暴露的货车的行为是可耻的：

“Where the most profit is made, the public is best served. Limit the profit, and you limit the exertion of ingenuity in a thousand ways. (来源: Newyorker - Liberalism According to The Economist)

不应该为了公众的福祉牺牲利润。工厂限制女人一天工作十二个小时是有害的。至于公共学校应该被取消，大众应该跟自己像是自己提供自己食物一样想办法给自己解决自己的教育问题。经济学人也是一样提倡：

“if the pursuit of self-interest, left equally free for all, does not lead to the general welfare, no system of government can accomplish it.

每个人追求自利就可以达到所有人的平等，如果这样都不能达到公众的福利的效果，那么也没有任何政府可以完成。他们强烈反对政府干预，但是一旦自由主义面对威胁他们要反过头来寻求政府的帮助。在19世纪的克里米亚战争，第二次鸦片战争和印度叛乱事件中英国的自由主义者似乎被吓到了。他们之前一直大力鼓吹的：自由贸易可以最大程度的最大效果的阻止战争的爆发。相反的，比如我们看鸦片战争，当自由贸易中对方不想跟你贸易时，这时他们会脱下所谓普世价值的伪装和把冠冕堂皇的信条抛到一边，求着政府说”我需要你的干预“，这个时候他们不在抵制政府或者说你政府别管，反而需要依靠政府的枪炮。

However, Britain’s expansion across Asia, in which free trade was often imposed at gunpoint, predictably provoked conflict, and, for The Economist, wherever Britain’s “imperial interests were at stake, war could become an absolute necessity, to be embraced.”

弗里德曼说的好，自由不是免费的，自由是需要实力支撑的，别人不要自由的时我要来裁决他需不需要，而军事武力是暴力的最直接最强大的体现。也能解释里根时代在减税和承诺缩小政府开支的同时还维持着巨额的军事开支来搞星球大战，以及在1999年弗里德曼的采访TAKE IT TO THE LIMITS: Milton Friedman on Libertarianism中提到什么可以取消时，他说了很多比如教育福利等等，但是军事是肯定不能的。

1860年代赫伯特·斯宾塞(Herbert Spencer)-一位自由资本主义哲学家-认为政府应该少干预自由市场竞争，同时认为政府应该少制定一些坏的法律(poor laws)-福利，因为这样干涉了进化的自然过程。他认为在资本主义的残酷竞争是好的，因为这样可以淘汰弱小的公司，这样可以可以促进社会的发展。政府不应该资助帮助那些在金融财务上挣扎的公司，因为允许这样弱小的没有竞争力的公司生存长期来看会伤害整个社会。同时他也把这个逻辑应用到个人上面，更进一步，他认为人类需要经历一场“净化过程Purifying process”来创造更能适应新工业化需求的更优良的遗传学上的人。优生学Eugenics当时已经是一个普遍存在的想法，但是斯宾塞对其特别热情，将社会达尔文主义和“survivalof the fittest适者生存”联系起来，应用在人类上。穷人的基因库最好能慢慢消失，因为他们显然无法在人类的下个历史阶段里适用更别说繁荣发展。

为什么说优生学是普遍的概念了？当时普遍认为其他人种是原始的低级的没有开化和缺乏理智推理能力的(1785年的托马斯杰斐逊Thomas Jefferson在佛尼吉亚笔记Notes on the State of Virginia 中描述黑人Blacks lacked forethought, intelligence, tenderness, grief, imagination, and beauty; that they had poor taste, smelled bad, and were incapable of producing artistry or poetry,黑人差于白人)，达尔文1871年在《人类的由来The Descent of Man》将自然选择理论应用到人类和引入性选择的概念。这里面就透露了他的白人至上的观念，他将澳大利亚人蒙古非洲印度南美等等称呼为savage未开化的人-非欧洲的白人的都是野蛮人，欧洲白人在可鄙视的野蛮人之上(highest races and the lowest savages)。两者区别在于:“moral disposition … and in intellect"（道德性情和智利思维逻辑能力）。白人有更好的逻辑思维和道德，其他野蛮人只有：“low morality,” “insufficient powers of reasoning,” and “weak power of self-command” （没有道德，缺乏理智推理的能力和自制力）.他怎么描述澳大利亚人了：

“How little can the hard-worked wife of a degraded Australian savage, who uses hardly any abstract words and cannot count above four, exert her self-consciousness, or reflect on the nature of her own existence”

动物不会思考自己从从何而来或者去哪里，什么是死亡或什么是生命等的。澳大利亚人缺乏复杂的思考能力，暗示他们跟低级动物一样。

“From the remotest times successful tribes have supplanted other tribes. … At the present day civilised nations are everywhere supplanting barbarous nations”

有了这个白人之上的种族理论和自然选择理论，达尔文可以正名化帝国主义的暴力行为：消灭非白人的种族是自然的选择，白人才是更优秀更成功的种族。

钢铁大王卡纳基在自己自传里写到自己读到达尔文和斯宾塞Spencer的作品时的激动心情时时这么写的：

“I remember that light came as in a flood and all was clear,” Carnegie wrote. “Not only had I got rid of theology and the supernatural, but I had found the truth of evolution. ‘All is well since all grows better’ became my motto, my true source of comfort.”

卡纳基在为什么这么激动了？因为斯宾塞展示了对于像他这样的男人强者在社会的顶端，这是非常正确而自然的现象，他这种资本家根部一点都不是邪恶的压迫的的坏人，而实际上正是推动社会前进的进步力量。卡纳基邀请史宾赛去Pittsburgh来看自己的先进资本主义的工厂。结果史宾赛相当的失望，因为他发现Pittsburgh这座城市污染遍地，噪音隆隆，热气沸腾，工厂充斥着压榨，资本家们过渡的迷信竞争，以至于他自己都惊了，评价这座城市: "Six months residence here would justify suicide.半年的居住可以证明自杀的合法性”（实在不是人待的）。

卡纳基怎么获得竞争的优势了？一部分是因为他残酷的压榨他的工人，让他们一天工作12小时，一周工作七天(997阿），只有伟大美国的国庆日才会放假一天。为了追求更高的生产效率可以放弃安全措施，让工人的生命处于危险之中。雇佣黑帮打手监工残酷镇压工会，工人有的列入黑名单有最后被杀害(当时工人的境遇跟奴隶有的一比，工会劳工争取自己权利的斗争历史相当血腥Labor Wars in the U.S.)。

1880年纽约的的政治漫画-描述了标准石油公司是一个horrible monster, whose tentacles spread poverty, disease and death怪物-充满了贫穷疾病和死亡。

弗里德曼自己在1973年的文章公平对自由Fair versus Free里写过：

Is then the search for “fairness” all a mistake? Not at all. There is a real role for fairness, but that role is in constructing general rules and adjudicating disputes about the rules, not in determining the outcome of our separate activities. That is the sense in which we speak of a “fair” game and “fair” umpire. If we applied the present doctrine of “fairness” to a football game, the referee would be required after each play to move the ball backward or forward enough to make sure that the game ended in a draw!

没有政府，自由竞争的市场谁来帮助是否公平了？他说公平在于建立一个普遍的规则和关于裁定关于规则的纠纷，而不是决定单个活动的结果。问题来了？他没说谁来裁定？是无形的手么？不太可能吧。是自由竞争中胜利的获胜方么？看起来比较像目前的情况！

弗里德曼说的很像之前我们描述过的1900年前的镀金时代的Unregulated capitalism不受监管的资本主义时代。自由市场当然留给了消费者很多选择，看似可以自由的选择，甚至过多，刺激消费主义，鼓吹这是一种个人主义的自由 - 选择的自由（vs之前政府没得选的自由）；对于在竞争中不太幸运的人不管是你的出生还是你的环境总之你就是被淘汰的那批人不够有效率efficient，那么就只能怪你不够努力了；但是没有人去思考造成这种现象”你很穷就是因为你不够努力“可能不是个人的原因，更多的可能是这个背后的系统性的剥削和不公平阻碍了每个人能公平的去获取资源。正如江总说的： “个人的发展除了个人的努力还要结合历史的轨迹”。这种系统性暴力带来的像是结构性的贫困，种族歧视和其他各种形式的歧视等等，更像是漂浮冰山水下的部分，而个人努力只是表面小小的一部分罢了。自由市场更适合创造财富，但是是不是更能分配资源了。市场是残酷的，与道德无关的没有情感的，尽管市场可能很有效率，但是那些出身不是很好的不是那么幸运的人就只能得到很小很小的蛋糕，过着挣扎的生活。

指望商业家不去欺骗和诈骗，有点天真了，只要看看洛克菲勒如何通过垄断的优势压价然后合并所有其他对手，最后一家独大的就知道，商业家门的动物精神可不是喝喝鸡汤就可以镇住的，孔子还说了-七十而从心所欲不逾矩-儒家修身内省时刻保持内心的警惕，不被欲望所裹挟到七十岁也才能达到这种境界。这些早期或者凭借条件率先从自由市场竞争中脱颖而出的人可不会轻易让别人来威胁自己的利润（max your profit by beating others)，他们可以花更多钱的来游说政府来通过更有利于他们的政策，这样互相加强不断循环。看看谷歌facebook推特等数字巨头们吧，推特甚至可以凭借自己的规则来封禁总统，facebook收购instagram和whatsapp强制用户共享隐私数据等等，苹果的内购30%分成被堡垒之夜控诉是大boss，谷歌威胁退出澳大利亚，这些实质已经是垄断整个行业。

另外贫富分化，垄断，全球化等等使得经济上美国传统的中产阶级被大大削弱，这些信奉保守基督主义的人看到自己先辈引以为豪的社区文化一点点被新自由主义的个人主义蚕食，经济上不得志，立足的文化信仰面临崩塌，不断涌入的移民和不断减少的白人人口，这些迫使他们拿起祖辈反权威反英英的传统，转向民粹主义，政治撕裂，社会撕裂，特朗普上台和之后的保守政策，到今年大选之前主流媒体预测90%拜登获胜结果出乎意料特朗普差点点赢了近一半的美国人支持他，国会山的动乱，正是新自由主义的反噬。

如果我们回到这个背景之下看美联储，可能可以稍微理解些。有一个Marshallow Experiment棉花糖实验:

在上世纪六七十年代，斯坦福大学心理学家 Walter Mischel 进行了一系列著名的“棉花糖”实验。他招募了600多名四岁的儿童参与这个实验。研究人员把他们带进一个房间，房间有一张桌子，桌子上放着一颗棉花糖。研究人员告诉孩子，自己有事情要离开一会儿，如果他们回来的时候，孩子没有吃掉棉花糖，那么就可以再得到一颗棉花糖作为奖励，如果吃掉了，则没有奖励。

结果是，每三位孩子中就有两位吃了棉花糖，有的孩子在房门关上后几秒钟就迫不及待地吃掉了，有的等了一分钟、有的等了五分钟、有的甚至等了十三分钟。而1/3没有吃的孩子，他们会看着棉花糖，不断往后推，甚至舔上一口，或者通过唱歌、踢桌子、闭眼睛来分散自己的注意力，直到研究员回来。

忍住短期诱惑不吃的孩子，往往要付出额外的精力和自制力来转移注意力-这个过程可见不是那么愉快的，才能获取到长期更大的回报。作为美联储或者政府，理论上理性上为了长期利益更多更好的回报，你应该短期内采取一些不那么愉快的行动或者痛苦的付出才行；但是如果这样，首先你不知道这个短期痛苦有多长，如果过长人民受不了这个苦-不是每个人都是那么理性的话，就惨了；其次你也无法明确给说明长期的好处具体是什么样的（奖励几颗糖），如果没达到预期怎么办；与其这样，还不如跟人民一起享受当下的了You only live once，正如凯恩斯说的“长期来看我们都会死”，所以还不如今朝有酒今朝醉，每个人都开心，至于明天会是怎么样，能拖多久，谁管它了，何必担心所谓的“死后洪水滔天”，也许它根本不会来根本就不存在 - 或者说经济学根本不应该拿来跟人做类比。

由此看来可能经济学家需要找到第三幅油门和刹车，至少目前来看两幅（财政和货币）看起来是貌似失灵了。

It worked great, until it didn’t.

自由主义低利率和政府的放松金融管制开启了金融化的时代，加上资本流动的全球化，技因特网的互联，使得价格和价值越加的脱离，经济指标和实际经济越来越像颠倒的对位，穷人和富人愈加的分裂。正如重商主义时的西班牙，钱货币正如亚当斯密说的应该只是流通的媒介润滑油，但是钱不等于国民财富，它的增加特别是不对应的顶部集中，是错误的。国家的财富是民众生产更多的产品来提高总体民众的生活质量。尽管现在生产的商品是够用了，甚至太丰富了，但是民众的生活质量很难说跟上了，顶部阶级让底层民众陷入在系统性的结构贫困陷阱中，中产阶级也慢慢缩小。记得中产阶级还是从底层工人阶级上升而来，被糖衣炮弹吸引分化的小布尔乔亚们可不会被忠诚空洞的口号等所左右，如果社会无法让中产阶级维持住他们的福利和利益，那么他们之前被遗忘的被埋没的革命斗争的火星可能随时被重新点燃。

当然资本主义的上层社会也不是静止不懂的，也不会坐以待毙，就像20世纪初自由主义者也知道在不断发展的工业社会，自由主义要需要缓和减速，之前被认为是绝对妨碍自由市场竞争和最大利润化的新政策 - 累进税收progressive taxation和基本的社会福利系统basic social-welfare system - 这些个代价是必要的，用来化解来自日益上升的工人阶级的不满。

总结来说我不是建议说去无脑梭哈房产、股票、或者比特币来对抗放水和资产价格水平的上涨，任何投资都还得是做好自己的功课do your homework - 最好还是个人努力结合时代发展的背景 - 保持开放的心态和好奇心扩大认知范围和过程的乐趣。任何时候都是高风险==高回报，反过头也是一样；如果有人推销说有低风险，高回报的事情，那一定是激起你的贪婪心，坑你没商量的；

且拭目以待吧。

John Locke洛克特别强调了财产（Property）的自由，但是他也对比了从原始状态到农耕到他那个时代，之所以出现不平等，他认为是贪婪，和货币的流通。好比之前你在怎么贪婪，也只能吃那么点猪肉，因为多余的猪肉你哪怕腌制着也会因为自然的腐烂无法长久藏很多数量，但是货币不同，你可以有无限的货币，随时随地可以兑换任何你想要的东西，特别是现在货币都不是实体流通的了，你有更大的自由去迎合人本性里最多的贪婪欲望，获取欢愉。但是洛克也特别强调克制，不能太过贪婪也不能浪费，未使用的财产是浪费的，是对自然的侵犯:

“因为任何人都可以在生活变坏之前利用其任何利益；他可能会通过劳动来固定财产。除此以外，任何东西都超过了他的份额，并属于他人。”

洛克认为财产、金钱和伦理道德是相互关联的，个人财富的累积和获得应该有moral伦理道德的限制（limits of accumulation）: «Enclosing in God’s Name, Accumulating for Mankind: Money, Morality, and Accumulation in John Locke’s Theory of Property»。

古典自由经济的代表人，资本主义之父亚当斯密的另一本著在《道德情操论》说的：

人类生活的不幸和混乱，其主要原因似乎在于高估了一种境况和另一种境况的差别。贪婪过高估计贫穷和富裕之间的差别，野心过高估计个人地位和公众地位之间的差别，虚荣过高估计湮没无闻和名闻遐迩之间的差别。受到那些奢靡思想影响的人，不仅在显示处境中是可怜的，而且往往为达到他愚蠢地羡慕的生活境况而扰乱社会和平。

“The great source of both the misery and disorders of human life, seems to arise from over-rating the difference between one permanent situation and another. Avarice over-rates the difference between poverty and riches: ambition, that between a private and a public station: vain-glory, that between obscurity and extensive reputation.

斯密提到了共情心共理心，在利己的同时，我们需要建立类似 “公正的旁观者” 的机制来自律，慎防出于自爱的动机异化成了自私而损害他人。保证利己不会因为自私和动物精神而失控。

问题是：理论上我们需要把握在在中庸的平衡的，不偏不倚，各方都好的理性人，完美；实际上动物精神本能驱动着在情绪的极端（贪婪，恐惧）中像钟摆一样来回震荡，而在中间那个完美的平衡点的时刻只是非常短暂和不稳定的。

借用韦伯的观点总结就是：形式合理性(工具理性)与实质合理性(价值理性)二者保持一致和谐的时间是短暂的，它们内在本质上不可避免的紧张对立矛盾意味中冲突才是长久的主题，不过人类作为先天具有理性的动物（总是尝试从动态变化的宇宙中观察希望得出可以预测的稳定的规律一样），总是希望得到一个静止的和谐平衡点，放开来看，这个平衡点是极其脆弱不可保持的。人的桎梏也是只是从之前的上帝换成了现代社会的商品机器 - 形式合理性和实质非理性的社会，用金钱和欲望的链条亲手打造一个个将自己囚禁其中的铁笼（Iron Cage）。有兴趣可以看看1930s年代的喜剧大师查理卓别林的具有讽刺意味的经典之作-《摩登时代》（ps:不自觉的联想到了近期PDD最近如厕压力-员工们除了高强度工作以外，括约肌也得承受巨大压力）。

“自从禁欲主义着手重新塑造尘世并树立起它在尘世的理想起，物质产品对人类的生存就开始获得了一种前所未有的控制力量，这力量不断增长，且不屈不挠。今天宗教禁欲主义的精神虽已逃出铁笼，但是，大获全胜的资本主义，依赖于机器的基础，已不再需要这种精神的支持了。……天职责任的观念，在我们的生活中也像死去的宗教信仰一样，只是幽灵般地徘徊着。……当天职观念已转化为经济冲动，……个人也就根本不会再试图找什么理由为之辩护了。”

清教徒想要成为职业人(Berufsmensch)——而我们则必须成为职业人。因为，禁欲已从僧院步入职业生活，并开始支配世俗道德，从而助长近代经济秩序的（虽然受到机械生产的技术与经济的前提条件所束缚）那个巨大宇宙的诞生；而这宇宙秩序如今以压倒性的强制力，决定着出生在此一机制当中的每一个人（不只是直接从事经济营利活动的人）的生活方式——而且恐怕直到最后一车的化石原料燃尽为止，都还是如此。

从天职和禁欲的“资本积累”（努力工作获取巨额财富和限制冲动消费来满足欲望），正如莱布茨尼等本打算用理性来更好的诠释上帝的存在，结果反倒是释放科学发展的洪荒之力某种程度瓦解了上帝的存在，希望求得通往彼岸世界的道路获得拯救（价值理性），现在没有了，没有彼岸世界了，再也不用恒长的警醒，不用克服自然的状态(苦行，一切都可以通过理性计算科学研究来预测推导（工具理性），而这个价值理性的空白正是由现实世界的享乐和不断膨胀的欲望的满足来填补, 新教的入世禁欲精神慢慢让步于纯粹的功利主义，成为了断了根的资本主义。

依巴克斯特的见解，对于外在事物的顾虑，应该只是像件披在圣徒肩上的“随时可以卸下的薄斗篷”。然而，命运却使得这斗篷变成了钢铁般的牢笼。禁欲已着手改造世界，并在这世界踏实地发挥作用，结果是，这世间的物资财货，如今已史无前例地赢得了君临人类之巨大且终究无以从其中逃脱的力量。如今，禁欲的精神已溜出了这牢笼——是否永远，只有天晓得？总之，获胜的资本主义，既已盘根在机械文明的基础上，便也不再需要这样的支柱。……没有人知道，将来会是谁住在这个牢笼里？在这惊人发展的终点，是否会有全新的先知出现？旧有的思维与理想是否会强劲的复活？或者！要是两者皆非，那么是否会是以一种病态的自尊自大来粉饰的、机械化的石化现象？果真如此，对此一文化发展之“最终极的人物”而言，下面的这句话可能就是真理：“无灵魂的专家，无心肝的享乐人，这空无者竟自负已登上人类前所未达的境界。(Fachmann ohne Geist(黑格尔), Genusmann ohne Herz)”

忘了从哪里看到的，说的很棒：人这种动物，一时半刻能够学会一点教训，中期会忘掉一大半，长期来看我们没有吸取任何教训。

[2020年关于现代经济的一些思考(六)]

2020年关于现代经济的一些思考(四)

2021-02-04T14:55:32+00:00

2020年3月份鉴于新冠疫情的原因美股连续几次熔断，美联储从3月15号开始首先开始常规的降息0.25%，降低贴现率0.0%，降低了对银行的资本要求，重启08年的QE包括购买长期国债和抵押担保债券Mortgage-Backed Security (MBS)等等。

如上图，存款机构（银行等）已经在美联储储备银行存了很多的钱了，这些钱机构并没有像美联储预想的那样流通到市场中去，而这些机构因为担心外面的风险等，反而更愿意存在美联储来获得利息。所以货币的传导机制出现了问题，但是美联储没有办法代替人家做决定，只能是间接的影响，而外面市场里则资金匮乏，大量投资者撤回投资手持现金坐上壁观，导致需要卖的没有了买家，而公司也面临着融不到资金周转困难，及其有可能恶化到影响日常的生产活动导致裁员降薪等等陷入更困难的境地。

美联储相比08年的这次的创新是直接跳过中间商购买公司的债券。建立了Primary Market Corporate Credit Facility (PMCCF)（一级市场公司信贷融资）用以支持新发债券和贷款融资 - PMCCF主要针对投资级公司，并提供4年的过桥融资；建立了Secondary Market Corporate Credit Facility (SMCCF) （二级市场公司信贷融资），主要购买投资级公司发行的二级市场债券，同时购买在美国上市的、投资范围是美国投资级债券的ETF基金。财政部将使用ESF对美联储为此设施建立的SPV进行股权投资 - 两项总计7500亿, 500+250Billion.

从2020年10月7号FED官方的总结报告《The Corporate Bond Market Crises and the Government Response》中提到背景：

In the U.S., when concerns over the coronavirus escalated in March, many financial markets were hit by extraordinary selling pressure. Bond mutual funds suffered net outflows exceeding $250 billion in March, or about five percent of their assets under management. Even funds specializing in short-term investment-grade bonds experienced outflows in March totaling eight percent of assets, dwarfing the selling pressure they saw during the global financial crises. Underlying these developments, liquidity provision by security dealers deteriorated, as some dealers reportedly reached their balance sheet capacity and hence were unable to absorb more sales. Together with the surging demand for liquidity, the pullback by dealers led to severely constrained liquidity conditions in the corporate bond markets. Amid the broad risk-off sentiment, corporate bond yield spreads widened dramatically (Figure 1), while issuance of new bonds came to a halt.

大量的人卖出债券换回现金，出现美元荒，导致公司融资困难。那么这些公司债券怎么运作的了？上面不是提到了一级和二级么？可以参考克利夫兰美联储分部《The Fed’s Secondary Market Corporate Credit Facility, explained》有解释和图表：

一级市场还是比较小而且量级比较大；二级市场比较广泛量级小，一级卖给二级；但是像上面总结报告里所说：

一级市场：But any improvement in funding conditions brought by the PDCF alone might not have been sufficient to induce increased liquidity provision.

Primary Dealer Credit Facility (PDCF) - If excessive selling pressure were to persist, dealers might be reluctant to take bonds into inventories given potential future challenges to turning them over.

On Monday, March 23, for the first time in its history, the Federal Reserve, together with the Department of the Treasury, created a facility to directly purchase investment-grade corporate bonds of U.S. companies in the secondary markets. Under the SMCCF, the Federal Reserve Bank of New York would provide recourse loans to a special purpose vehicle (SPV) that would purchase eligible investment-grade corporate bonds at fair market value.

一级市场有自己的顾虑，不像美联储想的那样，传导的效果有那么好；于是干脆跳过直接参与二级市场公司债券购买。我们看到3月份一开始美联储买的公司债券指的是要investment-grade(投资级别)。这个investment-grade指的是什么了？具体可以参考《What Does Investment Grade Mean?》.简单说评级结构根据公司的各个指标来划分这家公司的信用评价，比如上中下。“上” - 比如是苹果这种有核心竞争力，利润率很好，成长也很好的；中字幕意思一般；“下”则是比如公司可能是垃圾公司，有很大风险，没有稳定利润来源等等；那这些公司融资的成本也不一样，好的公司融资成本低，也就是债券的回报率低一点；差的公司就融资成本高，需要用高回报来吸引融资；investment-grade(投资级别)就是中上;非投资级别的叫做垃圾债Junk Bond.

4月9号《April 09, 2020 Federal Reserve takes additional actions to provide up to $2.3 trillion in loans to support the economy》，扩大到了在3月份22号之前是投资级别，然后3月22号之后信用下降到垃圾级别的的公司债券(称为Fallen Angel)同时还有那些暴露在以高风险高回报的公司债券为主要投资对象的ETF。主要目的是降低公司债券的回报率激增从而降低公司的借债成本（大量入等待卖出无人买，导致公债券没有要），最重要的是要显示美联储对市场稳定的坚定承诺，给市场以信心。随后确实比如垃圾债券市场的收益率大幅度下跌，公司融资成本显著降低。

直到5月12号美联储借助黑石BlockRock的专业知识来帮助其购买公司债券ETFs，几家主要的ETF比如先锋Vanguard/黑石BlockRock旗下的iShares。Fed Reveals Which Bond ETFs It Purchased有5月19号的详细的购买列表，引用其中一段：

Its iShares bond ETFs accounted for 48% of the Fed’s ETF asset purchases; Vanguard, 34% and State Street 15%.

Questions have been raised about the conflict of interest in the Fed’ hiring BlackRock to run several of its bond purchasing programs given the firm’s outsize role in the financial markets. Fed Chairman Jerome Powell told the Senate Banking Committee in May that BlackRock was hired for its expertise amid the Fed’s urgent need to arrange support structures for the bond market, which had been experiencing extreme volatility in March.

也许受到求助黑石BlockRock然后买了不少黑石BlockRock旗下的ETF引起了是否有利益输送的争议。2020年6月15号《Federal Reserve Board announces updates to Secondary Market Corporate Credit Facility (SMCCF), which will begin buying a broad and diversified portfolio of corporate bonds to support market liquidity and the availability of credit for large employers》美联储决定绕开黑石，总计成立一个公司债券组合，这个基于更广泛更多样的美国公司债券市场指数，Eligible Broad Market Index Bonds。

As detailed in a revised term sheet and updated FAQs, the SMCCF will purchase corporate bonds to create a corporate bond portfolio that is based on a broad, diversified market index of U.S. corporate bonds. This index is made up of all the bonds in the secondary market that have been issued by U.S. companies that satisfy the facility’s minimum rating, maximum maturity, and other criteria. This indexing approach will complement the facility’s current purchases of exchange-traded funds.

美联储买了哪些公司了？主要是汽车行业和科技行业。包括苹果，电信和几家在美国有分部有下属部门（subsidiaries）的国际汽车制造商。

大众，丰田，AT&T(美国电话电报公司-第二大电信供应商)，戴姆勒，Verizon威瑞森通讯，苹果，通用电气，宝马，微软，福特还有通用汽车，迪士尼，沃尔玛等等都还是比较出名的，后续还加入了可口可乐等等。美联储Jerome Powell 认为尽管目前借钱的成本已经比较低，但是鉴于新冠和快速恢复经济，还坚持购买公司债券，帮助公司需要现金来支付工人，坚定需要继续这么做，降低企业融资的成本。问题也看的出来，比如像苹果这种现金流很充分的特别优质的公司（现金奶牛），包括微软，甲骨文听起来不像是缺钱的公司。特别是苹果，在2020年8月09号宣布第三季度财报里显示，第三季度苹果的营业收入较去年同比增长11%，至597亿美元，本财年的前三个季度中收入增长了7%，达到2098亿美元；同时苹果在过去六个月（3-8）一直在不断回购股票（第三季度中回购了160亿美元的股票，全年回购的总价值达到了552亿美元，占该公司今年前三个季度所产生的运营现金的90%）。

坐拥2500多亿美元的现金的苹果公司还要发行债券融资。据悉，苹果计划近期新发行55亿美元的债券，这是该公司2020年来第二次发行债券。在今年五月份的第一次债券发行中，苹果筹集了80亿美元。和以往的债券发行一样，苹果计划将这笔资金用于股票回购和支付股息，以及其他“一般公司用途。”

拥有庞大的现金流，还要老发行债券来用于股票回购，说明在发行债券方面还是有利可图的。来自媒体的分析认为，苹果大量的现金是海外业务利润，存放在一些避税天堂国家，转回美国需要支付不菲的税金。而在美国债券市场融资利率走低的情况下，发债融资成为一个更好的财务选择，而不是调动海外利润现金。可以看出，即使是苹果公司，在涉及到现金流时，也是一直精打细算的。

过去几年中，苹果已经进行了价值数千亿美元的股票回购，这些操作导致上市流通的苹果股票数量减少，供求关系导致股价上涨，股东的财富增值。这恐怕也是苹果股价不断攀升，市值同步走高的原因之一

这篇《Why does Apple keep $250 billion in the bank when it has $88 billion in debt?》文章中非常具体解释了为什么苹果有那么多现金流依然要举债。

普通人认为有债务是不好的，巴不得立马还清，所谓“无债一身轻”，但这个常规的思路对于公司运营而言却是不一定成立的：债务反而是一种非常便宜的财务运作方式。如果用现金来支付日常运作，这些都是要收税的；但是债务的利息却是可以减免的；同时债务不需要跟股票equity一样，分享利润分红等，而且获得债券的需要的费用比equity要少；假设公司破产要清算，也会先卖出资产来支付债务debt，然后才是比如股东利益等equity; 公司不盈利不用分红，但是还是一定要支付债券的利息；另外苹果国外子公司赚的利润（这些钱已经在国外的国家被当地国家收过税了），这些钱要回到美国会再次征税，而且非常高，高达35%的税（ double tax foreign earned income）。

公司发行债券-就相当于公司负债，而负债部分在美国是可以减免一定税的(tax shield)，比如总统特朗普2017-2018年只缴纳了$750美元，很多人吐槽说总统都没自己交的多，其实因为特朗普那两年都是亏本的，债务抵消了部分税收。

这样当然有合理质疑是否美联储将钱用到了最需要的地方了。虽然大众丰田戴勒姆在美国有分部，想这些分部买债券确实合法合理，美联储更多的认为这些汽车厂商（比如福特）等等雇佣了大量的劳动力，一定需要优先包住他们的就业率。鲍威尔承诺一旦经济好转，会慢慢退出这些工具的使用，一开始承诺是到9月底，然后续到了12月底，根据FED最新2020年11月的发布，二级工具不会续期到明年了。

“美联储将会购买7500亿美元的公司债券”，“美联储会购买垃圾债券” 和“美联储购买了苹果和国外汽车公司的债券”这些连起来一起说，还是很容易让人觉得美联储的行为有些令人甚至愤怒。

我们看看到2020年底的这两项计划的实施情况：

三月份美联储宣称将购买7500亿，实际到了六月份也只满了55亿，到结束总共只买了130-140亿; 二级市场只买了50多亿
41%的公司债券评级为优A或更高，其他一半是普通BBB。ETF购买的债券部分有13%是高风险垃圾债券。

美联储这招还真是雷声大雨点小，情况也并没有说的那么令人愤怒。动不动就无限期QE或者像这样动不动数字巨大的购买，这个套路玩的很溜啊，按美联储的意思是：这体现了我们对于稳定市场的长期的坚定地承诺，直到经济情况好转（失业率和通膨率）。

截止到2021年1月，也有新冠疫苗产生出来的消息提振，公司债券回报率已经是大幅度下跌，甚至是里应该是高风险高回报的垃圾债券的回报率降到了历史低点(4.64%)，起码看起来企业债券融资的难度是大大降低了的。回到10月7号官方回顾《The Corporate Bond Market Crises and the Government Response》结尾部分：

While such a broad-based and speedy rebound in the multi-trillion corporate bond market might seem improbable given the relatively modest resources actually deployed, ideally, this is how a backstop is supposed to work. It restores market functioning by halting investor runs that can result in a self-fulfilling equilibrium involving serial defaults, widespread bankruptcies, as well as the collateral damage to defaulting firms’ employees and suppliers. Such an equilibrium almost surely would have deepened the economic contraction, causing more extensive damage to employment and economic output.

官方看来非常认可自己用小钱办大事的能力，这么点钱-一百亿就可以稳定甚至帮助几万亿的公司债券市场的反弹（从用词上看得出来有点小得意）。

表象和结果都说了，问题的起源了？在08年金融危机之后，利率一直很低公司的借钱成本本来就很低，从08年到19年美股十年牛市，经济看起来也还行，就跟农民也还都知道春夏天种粮食以保证冬天也有吃的基本道理，难道这些公司在日子好的时候一点点积存储蓄都没有么，以至于到了日子苦难的时候几乎没有啥存储现金来支撑熬过？十多年好日子(最长的经济扩张）撑不过几个月的苦难日子，难道是跟某个不谙世事的富家子弟一样挥霍了么?

2019年2月上任FED主席Janet Yellen曾警告因为长时间的低利息，不断增长公司债务压力，过渡的杠杆化借贷，可能在经济降速萎靡来时会引起深度的衰退。

“I have expressed concerns about leveraged lending,” Yellen said during a keynote discussion that was closed to the press. “I do think non-financial corporations have run up, really, quite a lot of debt.”

“What I would worry about is if the economy encounters a downturn, we could see a good deal of corporate distress. If corporations are in distress, they fire workers and cut back on investment spending. And I think that’s something that could make the next recession a deeper recession,”

借用知乎《Leveraged loan杠杆融资(以所借的钱作抵押所获之贷款)上的列子：

举个例子你就明白了：你现在身家1000万，但手里的流动资金为0，于是你向某些投资人发起私募（对冲基金），或者直接卖空，借到了1000万。你利用1000万引诱并迎娶（兼并或收购）了一个身家1000万的老婆，然后用你和你老婆的资质，在银行又贷款2000万。这一次融资就是杠杆融资。

利益放大，风险也放大了。2017年杠杆收购企业中，公司的债务是其收入的6.6倍，杠杆融资市场规模高达1万亿，1.2万亿垃圾债券，3万亿将将在垃圾债券评价上的投资级别债券（如果一旦下调，就是垃圾债Fallen Angel).

非金融部门的债券飞涨，比如上面的AT&T有1740亿债务，Verizon有1130亿债务。这些公司的债券就像08年的次级贷一样被打包糅合变成ETF在市场上卖给投资者。有些公司，通过低成本的借贷，上杠杆进行公司收购，用发生在将来美好的“利润”来进吸引投资者进行融资或者买入股票等其他高风险高回报的投资，同时将部分钱用来给高管CEO等发高工资，类似于击鼓传花的庞氏骗局。这些垃圾评级公司靠着再融资 - Refinancing (a method of paying a debt by borrowing additional money thus creating a second debt in order to pay the first - 靠着借钱来付债务)来左右腾挪其债务而避免其需要还清债务的义务（根本就达不到），而人们因为所有其他的有回报的投资的回报率都在普遍下降，所以人们更多投入相对高回报的债务投机中。

2019年11月的WSJ的文章《Corporate debt nears a record $10 trillion, and borrowing binge poses new risks》披露了：

Some of America’s best-known companies, including AT&T, General Motors and CVS Health, have splurged on borrowed cash.

“We are sitting on the top of an unexploded bomb, and we really don’t know what will trigger the explosion,” said Emre Tiftik, a debt specialist at the Institute of International Finance, an industry association.

El-Erian and others worry that an artificial environment of near-free money is masking serious underlying ailments and may be storing up problems for a future reckoning. This era of perpetually cheap money has kept alive some debt-ridden “zombie” companies that would have failed if rates were at traditional levels; widened the wealth gap between rich and poor; and distorted financial decisions, he said.

Lured by low rates, companies have splurged on debt to repurchase their own shares, pay higher dividends to investors and fund acquisitions, the IMF noted last month, contrasting those increases with what it called “subdued” capital investment. Corporations have spent more than $3 trillion over the past five years buying back their own stock, according to S&P.

就在2020年2月份在3月中美股崩盘之前的一个月中，这篇Junk bond scare is rising: ‘No one cares. People are buying everything’提到：

“It’s a risk, but no one cares,” Martin said. “People are buying everything.”

“A lot of the financial institutions are taking advantage and just refinancing and refinancing,” Martin said.

跟08年的次贷一样的，长时间跨度几年的不断上涨房价和长达10年（号称最长经济扩张周期）的美股牛市让人们似乎忘却了风险的存在，实际上的多米诺骨牌效应只要一旦触动就会是形势的螺旋式的升级，长很慢，跌很快。

而跟08年bailout银行等一样美联储在危机里出手bailout了这些公司。这些公司理论上如果按照自然流程，是会在危机里被清算，拍卖的拍卖来释放给投资的钱（虽然不多可能），倒闭的倒闭，要么处置不良资产或者资产调整结构重组，释放占用资源，短期可能有痛苦，长期看是好的，但是现在这些垃圾公司依然可以靠着美联储的这些救助勉强漂浮着（美联储的目的是打破多米诺骨牌效应，阻止水波想涟漪一样四处扩散，但是他也无法具体判断哪些公司影响小而可以倒，哪些不能，只能一刀切），占用了市场资源，使得那些具体竞争力有更好发展的公司没办法拿到更多资源，资源错配，影响了自由市场经济的那只无形的手的效用。这些企业就跟日本90年代那些银行是一样的，被称为僵尸企业或僵尸银行(zombie bank)，日本过了30年，很多僵尸银行在账目上依然有很大比例的不良资产。

这是截止2020年11月30号摩根整理的美联储应对新冠病毒的一些列措施的时间线，PDF的详细来源。

到2021月1月30日，美国的失业率从新冠从3.5%上升到三四月份的 14.7%，到现在大概是6%左右。3-4月份失业人群中40%的低收入人群失去了工作；纳斯达克从2020年三月的最低点到现在大概涨了94%将近一倍；新冠人数https://covidtracking.com//确证人数2千5百万，死亡42万，目前面临着疫情的反弹；

一月28号的美联储主席鲍威尔最新举行的发布会上FOMC Press Conference, January 27, 2021关于目前疫情的反弹中提到几点：经济的恢复继续取决于新冠病毒的控制情况;去年夏天经济活动的恢复反弹，在最近几个月变得有点缓和慢，主要集中在被新冠反弹和保持社交距离影响最大几个行业;总体而言经济水平还是在新冠爆发之前的水平之下，未来的道路也不确定；劳动市场也开始变慢，十二月份就业岗位下降了14万；还有数百万失业，经济衰退并没有平等公平的降落到每个人身上，服务业中低收入，非洲裔美国人和西班牙裔人收到的冲击最大；通货膨胀CPI去年夏天有快速上涨目前慢慢缓和，依旧低于2%的目标值；继续每个月购买800亿的国债和400亿的MBS直到经济好转; 经济恢复会是一个过程，充满未知。

在其中一个记者关于花旗银行研究说种族在收入和财富不均会拖累美国经济发展，请问美联储有什么措施来减小这种种族间的差距时，鲍威尔回到到他非常支持认可这样的说法，美联储的目的是通过最大化就业和实施公平放贷法来保证所有人可以分享美国经济蓬勃发展的好处。

理想的ideally, 很明显美联储认为放水有利于公司融资投资，这样可以扩大生产，招募更多的工人，进而带动更多人就业。

实际也不排除，鉴于失业率的下降，但是低收入人群的就业依然困难，特别是比如服务业里的餐馆和酒店，这些也是是提供就业的大头。还有实际的，这些大公司像是苹果通过低成本的发债融资，可以减免部分税金，同时用来回购公司股票，进一步推高股票价格，股东财富增值，这一招真是一举多得；而他们雇佣人扩大生产的积极性可能没那么强烈，雇佣的人可能没那么多，普通人低收入者很难从其中得到什么福利。我们看看大公司发行公司债券的成本有多低，比如苹果在2020年8月发行的10年期债券目前的收益率在1.16%左右，剔除同期预期通胀率1.72%，实际收益率仅为-0.56%。

当然鲍威尔也在2020年10月份说过Powell Warns of Weak Recovery Without Enough Government Aid：

“The pandemic is further widening divides in wealth and economic mobility,” Fed Chair Jerome Powell said Tuesday, warning that the country’s recovery will weaken without more government aid. “A long period of unnecessarily slow progress could continue to exacerbate existing disparities in our economy.”

他也意识到了新冠病毒进一步拉大了贫富差距，也让经济的流动性变差，呼吁政府更多帮助。

正如之前1980年代的涓滴经济学在减税开始的高税率初期，可能效果很好，但是当税率慢慢下来，接着减税可能就是得不偿失；现在也一样的，08年(几个月时间大概增加一万亿）之后又来百年难得-遇(2020年几个月时间大概增加三万亿）的放水，甚至有点泛滥了，自然这个效果也就难说了；

从1980年到现在，美联储获得了更多的自主权利，金融松管，配合科学技术的进步确实让经济获得很大的发展，但是也不是没有代价的。

贫富分化

根据美联储从1989年到2020年采集美国家庭财富分配数据《Distribution of Household Wealth in the U.S. since 1989》来看，到2020年第三季度，美国的家庭财富分配比例是这样：

底层50%的家庭只占总财富的%2；上层10%的家庭财富占比从1989年60.9%上升到了69%；特别富余的1%的财富占比人从1989年的23.7%上升到了30.%；而底层50%的人的财富占比从3.6%降到了1.9%。

这是不同财富阶级对于公司股票证券和共同基金的才有比例情况，右上角和左下角两个极值比较有意思。越穷的人股票证券这些资产越少，而且富裕的人持有公司股票或者基金等等的更多。最富裕的1%的人拥有超过50%的公司证券和基金，接下来9%的人拥有超过1/3的位置 - 也就是最富裕的10%的人拥有了超过88%的股份！

比较惨的还有Millennial Generation千禧年一代（从1981到1996年出生的人）大约有大约有7200万多人(美国总人口也就三亿多）大概是20%左右总人口，占据了劳动力的最大部分的构成，这些人却只控制了4.6%的社会总财富。

2021年1月30号最近的话有比较出名的wallstreetbets散户组团暴打华尔街巨头空头，甚至迫使华尔街资本控制的Robinhood居然限制了散户的只买不卖，这个事情甚至罕见了让美国民主共和两党的大佬都联合起来一致声讨Rohinhood就非常有戏剧性了。相当于华尔街基金们平时利用优势收个韭菜，结果没想到韭菜们集合起来有勇有谋的收割了华尔街，眼看就要破产了，就类似于拔网线搞这种无赖的操作，庄家在游戏中修改游戏规则，令人不屑。Robinhood本来是劫富济贫的，现在这个也太讽刺了。具体可以看看如何看待美国散户打爆华尔街大空头后被围攻，群聊被封，交易受限，媒体抨击，监管关注？此事后续会如何发展？和如何看待 1 月 28 日美股游戏驿站 GME 被 Robinhood 等券商禁止买入，导致股价狂泻？。

彭博社亿万富豪指数Bloomberg Billionaires Index列出世界上排名前500位富豪的总资产。今天2020月1月30号，目前Telsa马斯克超过Amazon贝佐斯20亿美元左右登顶全球最富裕的人，随后是MS比尔盖茨，LVMH的伯纳德·阿尔诺，Facebook扎克伯格，几乎前面都是科技类的。这一年里马斯克的特斯拉应该是最风光的，股票价格一路扶摇直上。世界上最富有的前10人在新冠期间财富增长了有5千多亿美元，主要得益于股票价格大幅度上升。尽管这些富人企业家因为自己超人的胆识，承受这不为人知的巨大压力和巨大的风险（马斯克后来披露特斯拉在2019年有一两个濒临破产，他本人自己因为在Joe Rogan的节目Joe Rogan Experience #1169 - Elon Musk上公开抽大麻和私人生活被批评，来推动技术的革新，理当获得相应合理的的回报，但是问题是这个数目合理吗？在疫情期间有4千万提交失业申请的美国人，2019年美联储自己出的报告显示甚至超过40%的美国人在银行拿不出$400现金美元应急。

富人变的越来越富，而这些现象伴随着美国大规模的失业率，严峻的新冠病毒形势，形成了鲜明的对比。这中间一定有什么问题的？

第一个原因是政府的救助方案失调，很大部分流进了大公司

回头看看，2008年经融危机中股市暴跌50%，8.8百万人失业，3百万人房子被银行没收，而从2009到2012年，底层99%的人收入只增长了0.4%而顶层1%的人却增长了惊人的31.4%. 当美联储不断注入货币和低利息，使得股市反弹，而中下层之前那些房子被收回失业的人手头已经没有钱可以投资进而获利，反而上层的有钱人手头有钱可以不断投入利滚利，回想下上面描述的公司利用低利息低成本的借贷不断发行公司债券，不断获取资金，投入回报更高的市场。想想看美股从2009到2019上涨了462%，一个一万块你可以拿到4.62万，如果是一百万那就是462万，这都没算上杠杆的各种套利，那将是指数型财富增长。

坎蒂隆效应(Cantillon Effect):

货币和信贷供应量的变化通过相对价格之变动而会对经济产生重大影响。货币增量并不会在同一时间等量地作用于所有价格，而是像在水里投下一枚石头一样，会一圈一圈地扩散，因而，增量货币会对经济产生什么样的影响，取决于货币注入的方式和渠道。增发货币未必会有利于所有人，相反，这可能会伴随一个再分配过程。先获得货币的人会推动商品价格的上涨，产生通货膨胀，而对于社会上的另一部分人来说，通货膨胀政策是对他们的一种掠夺。于是通过经济周期过程，完成了收入的再分配。

每当发生危机，钱总是先留到信用高的地方，然后一层层留到底层。谁会是信用高的了？大公司大银行有优质资产抵押的；谁会是低的？入不敷出没钱没房没稳定的工作，餐馆打工还有低收入的个体经营者；这些人在危机发生震荡时，大公司大银行有更多的缓冲；但是个人低收入者，特别是上面美联储自己的调查提到4/10都拿不出400美元的应急款，粤语中的手停口停，一点犯错的空间都没有，连生计都是问题；

我们看看2020年五月引发BLM运动的乔治.弗洛伊德事件的起因：

2020年5月25日傍晚20时许，弗洛伊德使用了疑似假钞的20美元在“Cup Foods”便利商店[注 1]买东西，店员察觉后拨打了911[27]。根据警方公布的通话记录，店员在报警时说，他曾要求弗洛伊德归还已经购买的香烟，但是弗洛伊德“拒绝这样做”；店员还表示，弗洛伊德“坐在自己的车里[注 2]，因为他醉得不省人事”，还“准备开车离开”[17]。

后面我查了下弗洛伊德的背景，更像是一个典型的美国黑人的贫困与暴力中艰难成长成长经历，这是没做成我们看到进入NBA的詹姆斯，过去陆陆续续被毒品偷盗持枪抢劫的犯罪记录。但是在这次事件之前弗洛伊德辗转了几个州尝试不同的工作，看来应该向想尝试通过个人奋斗教育工作来脱离暴力的原生环境，来争取女儿的抚养权，在其他人帮助下他也成功戒毒，在当地餐厅和夜间音乐俱乐部打两份工。而今年席卷全美的新冠疫情，在佛洛伊德身上添上重压。他丢掉了两份保安工作，成了疫情期间数百万失业美国人中的一员。时代的一粒沙，落在个人身上就是一座山。你无法苛责在困境潦倒挣扎的人做出似假钞的20美元在Cup Foods便利商店,更难以简单用过去的暴力犯罪经历就一笔抹掉后面他想摆脱贫困暴力想回到家庭所做的努力，尽管一再收到生活的打击。反而你要问的是为什么是这样了？这背后反应了什么深层次的问题了。

btw: 少数精英亚裔在硅谷和精英阶级里的成功使得很多人幻想都可以成为光彩亮丽的上层名流，但却忽略了亚裔正是美国贫富差距最大的群体，大部分亚裔是默默无闻的中餐馆啊或者底层劳动的苦命人，过着贫苦美国梦，甚至英文都不会说完整；而部分亚裔精英可能觉得只要更富有更有知识进入曼哈顿或者硅谷，忽视甚至鄙视这些自己族裔的贫苦之人，放佛他们成为不了美国人的原因就是因为他们太low贫穷没知识愚昧，巴不得与之绝缘，这样好像自己就可以被接纳为纯正高级正统的美国人，甚至某些因此憎恨自己的黄皮肤，不够白(不禁想到乔治卡林Geogry Carlin的"Colin Powell is openly white - he just happens to be black")，反思亚裔如何才能更爱国如何更模范，通过妥协compromising&sweet talk来取悦白人换取认同（比如杨安泽在华盛顿邮报的文章《Andrew Yang: We Asian Americans are not the virus, but we can be part of the cure？》中提到路人的异样的眼光让他对自己身为亚裔感到“有些羞耻”, 亚裔效仿二战中被拘留的日裔，用爱国行为自我牺牲来证明忠诚，他本人更是以被强加亚裔数学很好的stereotype刻板印象作为竞选的口号），且不论想法之幼稚（固然矛盾的身份identity带来的冲突矛盾可以理解），事实是，简单的说，白人黑人看到你亚洲人歧视你仇恨暴力袭击你是不会先开口问你是日本韩国或者自认娘家的所谓的亲民主派的台湾香港人，还是大陆人，亦或是上流社会有多纯正ABC还是刷盘子的不会说英语的(看看这位老人吧，‘Is this patriot enough?’: Asian-American veteran shows scars as he calls out anti-Asian hate看着可怜但又说回来这要能说明什么，为什么黑人白人不用展示自己的爱国和忠诚了，这么说吧，即使亚裔比如杨安泽在自己的前胸后背都纹身纹上I hate being Asian/I am 100% American，不穿衣服上街，一样照样被歧视Asian American Air Force Veteran Attacked in Koreatown)。

美国常说是民族的大熔炉，包容各个民族的不同，这话某些情形是对的，但是也意味着你不要忘记相同和融合的前提是不同，过度强调同，甚至忽视本身同的前提是本质的不同，甚至拿这个都成普世价值不变的真理就有问题了，当形势和背景发生变化时，是不是一直都会是同>不同了？经济形势好的时候，人人都还可以，大家还可以忍住自己人性的兽性本性，维持着所谓的道德和普世价值，标榜鼓吹自我优越（John Locke洛克-自然状态人性善）；当形势不好的时候，就可难说了，历史上的案列太多，什么所谓的道德标榜，还不是回到了丛林法则(Thomas Hobbes霍布斯-自然状态人性恶），每个人都想以邻为壑，发泄压力（再加上特朗普不断的用中国病毒来重复洗脑乌合之众，久而久之成为了大众的潜意识，这种状态下理性让位于生存的原始欲望感性），总的挑一个软柿子来作为替罪羔羊吧(scapegoat)，这里面组成的不同的成分哪些是比较弱的？一边是软柿子，一边是带刺的，选哪个了？亚裔内部派系林立，一团散沙，各有各的想法目的（不像黑人没有非洲可以回的去了），也只有当亚裔普遍面临生存危机的时候，才能有偶尔零星的团结，可以预见的是风头过去之后，还是老样子。

来源: https://thomasnastcartoons.com/

杨安泽对于自己族人-华人和亚裔的历史的知识更是相当匮乏的，他可能都不知道王清福这个人。如果我们回顾下华人在美国的历史，这里我就拿有”美国政治漫画之父”的托马斯纳斯特Thomas Nast在1871年在Harp Weekly上面发表的“The Chinese Question”- (Columbia.–“Hands off, gentlemen! America means fair play for all men.”)的漫画看看吧。这个背景当时大量的猪仔或是被骗或是因为逃难亦或是为了更好的生活离开家人追梦到了旧金山，大量的廉价的来自中国的劳动力进入美国，跟之前1830年-1840年之间因为饥荒和战乱等等逃到美国的信奉天主教的爱尔兰人和德国人，发生了竞争，引起了爱尔兰人的极度不满，抢了他们的工作饭碗。爱尔兰人刚到美国时候跟中国人一样，都是从事被当地人鄙视的看不起的底层工作，现在以爱尔兰文化出名的波士顿当时还发生了针对天主教爱尔兰人的迫害，这些引发了爱尔兰人对如何保护自己的思考，还的是要投票要有政治权利和影响力，通过人多齐心投票来争取自己族群的利益，改善了自己的地位。但是这些人看到后面一大波中国人过来抢了他们工作，比如太平洋铁路，做的比他们还好价格比他们还便宜，立马就通过自己刚争取到的政治权益打压更弱势的中国人，全然忘记了自己当初刚到美国时时怎么受到迫害的，最臭名昭著的就是爱尔兰移民丹尼斯Denis Kearney每一次演说结束都会煽动”无论如何，华人必须离开”结束。(“And whatever happens, the Chinese must go”），对于华人的仇恨运动anti-chinese也是越来越升级，口头上更加直言不讳，行动上也越来越暴力，成效也是越来越成功，两党也是争相响应”民意“，1882国会还是通过臭名昭著的排华法案Chinese Exclusion Act 。

纳斯特这幅漫画就是在此阶段做的，立场不一定是他同情中国人，主要是他本人对爱尔兰人实在太厌恶了。建议放大看看此图，这幅画里女神左边的有对华人的大量的诽谤，被称为野蛮人、异教徒、未开化、不诚实没道德的、反对家庭、卑鄙邪恶之人，通过这些名称标签将个人的特性人性（individual identity）抹掉，替换为模糊笼统的群体集体名词，比如类似现在的污名化的Chinky，当时不叫这个而是图里面的John Chinaman。而有意思的是女神右边那些武装mob暴徒，为首的右边起第二个就是爱尔兰族裔的美国人，最右边应该是个德国裔美国人，还有一些其他暴徒，持有抢和武器，时刻准备用武力暴力威胁手无寸铁的华人劳工。图里面还提到工会会议决定必须通过ballot投票或者bullet子弹来来组织这些野蛮的华人输入。

对比现在，惊人相似，黑人是爱尔兰人，亚裔就是当时的华人。尽管黑人历史上被白人奴役，经过黑人民权运动等等抗争在平等性已经有了很大的进步，但是对于比自己人还少还能忍气吞声的亚裔，情况是什么了？ BLM运动亚裔支持黑人运动，理应两个弱势群体应该联手争取更大的权利，但是情形一旦不对（经济形势恶化），亚裔曾经认为的盟友-黑人，发现被主流媒体称赞的所谓的模范的亚裔（像上面所讲其实亚裔的贫富差距是最大的，少数亚裔的光环被主流媒体刻意树立起来当成了靶子）的收入学历教育居然比自己还要高那么多（好比爱尔兰人发现华人劳工钱少活好），带刺的不好弄搞不好伤了自己，于是反过头来就着柿子就是一刀，发生似曾相识的爱尔兰人疯狂针对华人的一幕也就是自然而然的了。有意思的是，袭击亚裔的老人和妇女中主要都是黑人，但是主流新闻一概不描述其黑人的身份，所以啊还是要闹才能争取自己的政治正确。

我可以理解感受亚裔想要融入的心情，毕竟每个人也都想过好的自己生活不是么？这个过程中身份认知造成困惑，感觉始终没有被真正接纳的沮丧，融入是不是意味着抛弃自己的身份认知了，甚至部分极端地鄙视憎恨自己受之父母的身体发肤了？

希望亚裔还是多了解点历史，多学习点黑人运动，会哭的孩子有奶喝，回归集体主义，该团结的时候还是要团结下的，正如1962年Malcolm X in Los Angeles May 5, 1962 Who taught you to hate yourself? full speech说的好：

“Who taught you to hate the texture of your hair? Who taught you to hate the color of your skin, to such extent that you bleach, to get like the white man? Who taught you to hate the shape of your nose and the shape of your lips? Who taught you to hate yourself from the top of your head to the soles of your feet? Who taught you to hate your own kind? Who taught you to hate the race that you belong to, so much so that you don’t want to be around each other?”

回到正题，当经济恢复时，往往有着强大投资和购买力的富豪们，借助于政府的资源和自己的资源，可以在其中获得巨大的利益。关于这中间财富分配和资源分配我们可以借助于概率论，比如杨辉三角或者说高尔顿板。

来源：Galton board form PhysicsFun: https://www.instagram.com/p/B5qLb2DlZPA/?hl=en。

可以借助于杨辉三角理解下，到达中间的球的路劲比旁边的位置的要多 - 这样代表比如都美联储在上面放水时候，他设想的是每个位置获得差不多的理想情况，但是实际可能是：

来源: https://twitter.com/cwalsh516/status/1095026717184352256

信用级别：政府 > 大银行大金融机构 > 大公司大企业-> 大于个人或小微企业。美联储放的水经过企业家银行家也都留到了股市中，特别有意思的是由于Robinhood等面手续费的交易平台的兴起，以WallStreetBets为代表的的这群千禧一代的散户们，也积极投入到了股市的投资或者投机之中，挑战华尔街的金融规则，他们被华尔街这群建制派称为异类被他们鄙视，但是也进一步让美国股市异常火爆。

IPO & SPAC

2020年的技术类的IPO公司数量超过了2001年，比较出名的饿了么DoorDash和传统的Airbnb，Snowflake等在第一个交易日就几乎都涨幅超过100%，这三家公司是最大IPO价格增长的前10名 - 也就是说股票上市前的定价在第一天交易就被追捧，上涨幅度之大令人咂舌2020Dec IPO Mania Sweeps Over Robinhood Crowd and Stokes a 111% Rally。

举个例子有多疯狂，Airbnb上市定的价格是68，老板Brian Chesky在接受Bloomberg电视节目直播采访时候得知第一天上市交易股票价格翻倍时(收盘时价格$144)，他甚至震惊的一度支支吾吾哑口无言（momentarily speechless）。他怎么说的了？

That’s the first time I’ve heard that number. Um, that is, that’s … I … you know, when we … in April, we raised money, it was a debt financing … uh … that price would have priced us around $30. So I … I don’t know what else to say. It’s … that’s a … that’s a very … that’s … um … that is … yeah … I’m very humbled by it.

很明显Airbnb根本没想到股票价格居然上涨幅度那么大，如果你想想基础面 -AIRBNB的业务场景和新冠病毒的形势，大部分人认为是Robinhood的散户推动的(所以说WeWork2019年撤回上市太亏了，如果在2020年绝对能上市，软银太惨了）. 所以华尔街的投资公司的经理Michael Holland, chairman at Holland & Co认为这群散户疯了:

“You simply have to use common sense and step in and say, ‘we’re not going to participate in this craziness,’” he said in an interview on Bloomberg Radio and Television.

除了传统的IPO，还有一种公司上市特别受到散户的欢迎就是Special Purpose Acquisition Company (SPAC)-特殊目的收购公司 .从这篇介绍的《SPAC 是什麼？有什麼好處？運作流程與結構？SPAC IPO 上市為何在美國爆紅？》的文章我摘录下：

「特殊目的收購公司」（Special Purpose Acquisition Company，SPAC）是一種沒有營運業務的空殼公司，唯一的目的是透過首次公開發行（Initial Public Offering，IPO）募集資金後，專門去收購有前景的未上市公司，等同讓被併購的公司能夠借殼上市；換句話說，SPAC 是一種讓私人企業能夠「借殼上市」的公司。

也就是說，這個「特殊目的」指得就是「併購未上市公司」，而 SPAC 的運作流程就像這樣：成立一家空殼公司（SPAC）» 透過 IPO 募集資金 » 尋找有前景的未上市公司 » 併購目標公司 » 等同讓被併購的私人公司立刻上市 » 股價漲漲漲 » 投資人和被併購的公司都開心（如果沒有意外的話）。

SPAC -「特殊目的收購公司」為何又稱為「空白支票公司」？ SPAC 在申請 IPO 募集投資所需的資金時，其實還不確定會併購哪間公司，要併購的目標公司是待 SPAC 經由 IPO 募集完資金上市後，才由 SPAC 的管理團隊物色適合的併購目標；因此投資人購買 SPAC 股份的當下，並不知道自己將會投資到什麼公司，只是先瞎挺管理團隊，就像開出一張空白支票，所以 SPAC 也被稱為「空白支票公司」（Blank-Check Company）。

SPAC 的管理團隊會不會遲遲不併購公司？ SPAC 必須在一定的期限內尋找到併購目標，並完成併購。否則，SPAC 將被迫清算解散，將信託資產返還給股東。通常 SPAC 從 IPO 結束到完成併購，必須在 24 個月之內完成。

简单的说可以不总传统的IPO流程（很多钱和上市原始股份给了投资公司），可以节省费用，跳过繁琐复杂的审计流程，主要是通过有名声的管理人（与之对应的是管理人可以以很便宜的价格持有相当比例的原始股份）来寻找有潜力的私人公司，帮助其快速上市（90天对比18个月），实现双赢。这种SPAC IPO在2020年彻底爆发，根据纳斯达克2020年12月28号的统计显示SPAC的数目增长了4.6倍到239家，募集了超过790亿美元。没有监管等等开放透明的资料，管理人的信誉和背景就是非常重要的。SPAC中比较出名的管理人就包括Bill Ackman和前Facebook高管查马斯·帕里哈皮蒂亚Chamath Palihapitiya。Chamath Palihapitiya一直是SPAC最多产的赞助商之一，将其与一系列公司合并，包括太空旅游公司维珍银河Virgin Galactic.

维珍银河航班的价格约为每人25万美元，该公司目前已经有603名意向客户。据瑞银集团（UBS）预计，尽管太空旅游业仍处于“新生阶段”，但十年内其潜在市场规模有望达到30亿美元

他本人投资过比特币，并和他人在油管上公开辩论，今年的比特币飞涨也让他的声誉高涨， Social Capital的CEO，金州勇士的老板之一。Chamath Palihapitiya初始股东可以以价格是$0.002美元每股的价格持有20%的股票，其他在市场的投资者只能用$10美元的价格购买剩下的80%.

从2020年九月份2020-Sep - Chamath Palihapitiya unveils $4.8 billion SPAC deal for real estate startup Opendoor他的方向是进军房地产行业，简化和方便流程，目标是围绕着千禧一代，他的目标是找到类似亚马逊和特斯拉相似的可以复制的商业模式。

我个人觉得这个虽然听起来很不错，感觉总是有那么点太美好了（直接绕开狡猾腐败的华尔街，寻找到最需要资金的下一个地下室的苹果，帮助私人小公司融资投入更多的研发），潜在的利润可以线性推测，但是现实的发展往往不是线性的，复制成功的商业模式也太难了，另外特别是你现在处在一个泡泡里，一旦形势不太好，没有合适的投资对象，那么就是另外一个话题了，是不是投资者能拿回钱了。过去的成功和巨大的声誉不一定能代表未来的成功，这些因素太难预测了。当然这个是暂时个人的观点，可以看看Motley Fool的文章《How to Invest in SPACs》.

总体而言多一种融资和上市的渠道，也是对小企业小公司发展而言也是一个好消息，其他的留给时间看看。不出意外Chamath Palihapitiya也在最近的Gamestop和AMC事件中为广大散户呼喊，挑战华尔街的建制派。背靠大佬支持的散户大战空头的戏码，与本月特朗普的支持者冲击国会山遥相呼应，一个在线下，一个在线上。两个事件有一个共同点，那就是底层群众对精英阶层的宣战，只不过这次从政治精英变成了财富精英。还记得2012年的占领华尔街运动么？结合起来看广大散户人民真是苦秦久矣。散户们的疯狂只不过是对美联储放水和既得利益者华尔街精英派的控诉罢，问题表象看起来是对散户们的道德拷问妖魔化散户（比如Melvin Capital雇佣CNBC媒体抹黑这些散户），结果确是对华尔街建制派的灵魂追问，散户们历数华尔街的罪恶（08年金融危机投机给群众带来损失，没有惩罚结果被救；接着利用机构优势非法做空，还雇佣媒体抹黑，操作交易平台robinhood等还限制购买等等）。

1962年Malcom X在Malcolm X’s Fiery Speech Addressing Police Brutality说的就是你CNBC：

“The media’s the most powerful entity on earth. They have the power to make the innocent guilty and to make the guilty innocent, and that’s power. Because they control the minds of the masses. The press is so powerful in its image-making role, it can make the criminal look like he’s the victim and make the victim look like he’s the criminal. … If you aren’t careful, the newspapers will have you hating the people who are being oppressed and loving the people who are doing the oppressing.””

当然这个也可以套用其他新闻上比如BLM运动上，主流媒体很快将关注点聚焦到抗议运动的打砸烧上，很快能引起人们对这群”乌合之众“的憎恶，进而开始怀疑这个运动本身的正义性。但是他们没有关注到相对于大规模的全球性的相对和平的抗议相比，这些零星的破坏性活动其实相对而言是比较少的，但是通过这些局部的非正义（因为迫害太深的反弹和零星的害群之马叠加）的大肆报道可以抹杀掉整体上面的正义，激发广大群众的反感，冷却下好不容及激起来的人性共情的情绪，这样大家就会对这个活动失去支持和兴趣，让这次事件和抗议又一次的无疾而终-缺乏上下文的事实有时候比谎言还可怕。总之wsb vs wallstreet对战后面接着吃瓜看戏。

从1980年新自由主义兴起之后的美国社会财富分配，引用达里奥的债务危机中的图片：

贫富差距在拉大，当然不当是美联储放水导致的了，还有很多其他因素，科技金融创新，全球化进程等等。但是看得的出来美联储救市应该说并没有劫富济贫，反而是有些拉大财富差距的促进作用。弗里德曼的新自由主义在1970年之后特开始有影响力，到了1980年之后被广泛的实施，他所在芝加哥大学的部门后来出了30个诺贝尔经济学奖得主，也使得他是最有影响力的经济学家。弗里德曼的新自由主义跟目前的贫富差距有没有直接关系了？

1970年弗里德曼发表了《The Social Responsibility of Business is to Increase its Profits 商业的社会责任就是增加利润》被批评者认为是开启了Greed is Good贪婪是个好东西的经济自由时代，追求利润是商业唯一的真实目的；其他人认为它是一份及时的资本主义宣言-清晰的描绘了商业高管们在自由市场应该有的角色。在当时看来，商业应该在追求利润之外还有其他的社会责任 - 就业，跟歧视斗争增加平和避免污染等等。弗里德曼则说到：

“Businessmen who talk this way are unwitting puppets of the intellectual forces that have been undermining the basis of a free society these past decades,”

弗里曼的认为作为公司的高管或者CEO，他的身份只是员工，股东是老板，那么CEO的唯一目的就是在遵守基本的社会规则的前提下，给老板挣得更多的钱。不以为公司挣得越多越好的CEO是不称职的员工，做社会责任可以以个人身份做，但是作为员工必须是紧贴满足老板和上级的欲望。

“This is the basic reason why the doctrine of ‘social responsibility’ involves the acceptance of the socialist view that political mechanisms, not market mechanisms, are the appropriate way to determine the allocation of scarce resources to alternative uses,”

不符合自由市场竞争资源优配的原则。

But the doctrine of “social responsibility” taken seriously would extend the scope of the political mechanism to every human activity. It does not differ in philosophy from the most explicitly collective doctrine. It differs only by professing to believe that collectivist ends can be attained without collectivist means. That is why, in my book Capitalism and Freedom, I have called it a “fundamentally subversive doctrine” in a free society, and have said that in such a society, “there is one and only one social responsibility of business–to use its resources and engage in activities designed to increase its profits so long as it stays within the rules of the game, which is to say, engages in open and free competition without deception or fraud.”

这个在今天还是非常有争议的。现在很多企业现在也都会鼓吹就业平等，包括种族平等性别平等，很多公司比强生也都会把客户的利益放在第一位其次才是股东的利益；其次弗里德曼的理论只在完美的市场场景（没有欺骗和欺诈的公开自由的竞争 - 这个可能么？）下发生；很多CEO的薪水相对普通员工相比简直是指数式的差别(320倍-CEO compensation surged 14% in 2019 to $21.3 million - CEOs now earn 320 times as much as a typical worker），他们貌似不管公司是否盈利给股东以好处，更多是自己的利益；大公司追求自己的利益花了大量的钱游说国会放松监管(请看2018年amazon亚马逊赤裸裸地威胁要停止建设新的大楼，成功说服了西雅图市长放弃一项来帮助城市流浪者的税收政策Amazon Just Killed a Tax That Helps Homeless People）；

America is out of balance. CEOs earn 320 times the average worker. The top 10% of American households owns 70% of the country’s wealth.

08年经融危机中倒闭或者被救市的企业的高管依然拿着高额的薪资，而讽刺的是救助这些企业的钱却是普通老百姓来买单；这一幕有点类似资本主义早期1842年罗思柴尔德伯爵夫人佩戴价值150万法郎的珠宝出席奥尔良公爵的化妆舞会时，数以千计的饥民却从窗外走过的场景，甚至比这个更恶劣。

再看一下按照弗里德曼的解释，那么美国目前社会的撕裂，应该是他希望看到的。为什么了，企业为了追求利润比如追求回报率更高人工成本更少的地区，而全球化正给了这样一个舞台，美国很多公司将制造业等等转移出人工成本昂贵的美国到东南亚和中国这种人工成本低的国家，这样一来，美国最能承载就业的制造业出现了空心化，比如五大湖的铁锈地带。这个去制造业和金融化的过程，使得利润源源不断流入美国的华尔街，大公司和高管，而之前蓬勃的中流砥柱的中产阶级面临着经济的苦难和传统核心社会文化的奔溃。这个理论看起来就是社会达尔文主义的经济版啊。

客观而论，我们要考虑弗里德曼当时的历史背景，当时政府的管控特别多，监控和流程也是非常繁杂，确实一定程度造成了资源的错配，影响了经济的发展（还记得尼克松和后来福特卡特的工资物价控制么），高达70%的税率，政府的手勒的太紧；弗里德曼敢于在70年他有影响但是没有到后面语言滞涨得到验证时那种成为主流的影响力，敢于出来表达自己挑战一般传统的观点（僵化凯恩斯主义），还是值得钦佩的；但是后续自由市场发展打来的贫富分化不晓得是不是他也考虑到的了，要么太紧，要么太松，这个怎么讲了，只能说： ”没有绝对对错，凡事得有度“。

贪婪本身定义就是模糊的，没有具体量度的，一切事物对一个人的价值只体现于多大程度满足了其自身的需求上，而这个尺度是主观的是难以把控的。特别是在不精确的经济领域，甚至哪怕在精确的科学领域，也没有银弹，哪怕爱因斯坦也都不是都是对的他自己也承认自己会犯错，所以没有一种理论或者框架是代表了自然的至高真理，所以说自然才是永恒的最高真理的代表人，任何人都只是短暂某个时间段里的代言人罢了，如果哪天狂妄自大不知所以的话，自然定会给他好好上一堂课的 - 弗里德曼可没有提过亚当斯密的的《道德情操论》-一本亚当斯密可能觉得比后来的国富论还更重要的书了。

抛开美国，看看弗里德曼有趣的实验的智利，曾是被芝加哥男孩称为”经济奇迹“，被认为是新自由主义在拉美的标杆。大量的国企被私人掌握，官员特俗集团和外国势力手中（总统皮耶罗就是一种一位），智利的贫富差距难以相信的程度，2019年因为智利首都圣地亚哥的一价制地铁票涨价三毛，引发了巨大的抗议浪潮，成为压垮骆驼的最后一根稻草。这些地铁，公交车，电力，水力包括医院等资源都掌握在私人手中，按照在社会规则下追求利润最大化的要求，他们一起不断地涨价，使得普通人的日常生活负担越来越大。

有兴趣的可以看看The Fall of Chile:

The neoliberal experiment—in Chile—is completely dead. It is likely to be replaced by a welfare state that will attempt to follow the Nordic countries. —Sebastian Edwards (2019)

迫于压力，这位从1980年随着新自由主义兴起跟着引入信用卡的和其他投资发家的身价28亿美元的总统皮耶罗是这么回应的：

It is true that the problems have not occurred in the last days, they have been accumulating for decades. It is also true that the different governments were not able to recognize this situation in all its magnitude. This situation of inequity, of abuse, has already meant a genuine and authentic expression of millions and millions of Chileans. I recognize this lack of vision and I apologize to my compatriots.

拉丁美洲的魔幻现实主义，借用地球知识局这篇关于解放神学运动文章里的一句话：

富人在优越的自然资源支持下纸醉金迷，穷人在恶劣的通货膨胀中抢夺最后一只面包；保守的主教在高台上接受信徒的朝拜，愤怒的革命者在街头丢出手里的最后一块石头。

写《新教伦理与资本主义精神》的马克斯·韦伯怎么看新教对于资本主义发展的促进了 - 从奥古斯丁的原罪《上帝之城》《忏悔录》（中国古有孔融让梨美食，西方有奥古斯丁偷梨，奥古斯丁怎么看待自己为什么要偷梨了，不是为了吃，就是心痒手贱无聊刺激，跟亚当夏娃一样上帝什么都给了就是不安安稳稳的，所以人生来就带有罪恶啊 - 天主教的罗马教宗是上帝在俗世的唯一代理人）到马丁路德的“因信称义”再到加尔文主义的“预定论”，教徒不再是被动式的在充满罪恶的俗世消极的处世（超尘出世Weltfremdheit-只能通过忏悔和参加教会的仪式等待才能洗清自己的原罪，才可能死后上天堂，但是谁也不知道自己是否至善是上天堂-少数，还是至恶下地狱-少数，还是大多数中间的炼狱,可以参考14世纪的但丁《神曲》中宇宙的结构 - 所以有了赎罪券，类似于“花钱买功德”）

ps: 1095年，在岌岌可危的东罗马帝国面对着博斯普鲁斯海峡对岸气势逼人的塞尔柱军队时，罗马教宗乌尔班二世在今天法国的克莱蒙召开了宗教大会，发表了著名的极具煽动性的演说，拉开了十字军东征的序幕：

本着主赐予我的权柄，我郑重宣布：凡参加东征的人，他们死后的灵魂将直接升入天堂，不必在炼狱中经受煎熬；无力偿还债务的农民和城市的贫民，可免付欠债利息，出征超过一年的可免纳赋税。

凡动身前往的人，假如在途中，不论在陆地或海上，或在反异教徒的战争中失去生命的，他们的罪愆将在那一瞬间获得赦免，并得到天国永不朽灭的荣耀。

向着东方出发吧！不要犹豫，不要彷徨，为荣耀我主，去吧！

到了加尔文主张上帝已经根据自己的逻辑选好了一部分人上天堂了，是他的子民（预定），在俗世你干什么都不能改变你是否死后上天堂还是下地狱或者还是中间熬死人的炼狱，这个上面教会等等都没有权利。听起来还是很消极，还是一种出世的概念，既然都是追求死后的上帝之城，那么现实世界还有什么意义了？加尔文这里就很巧妙的将上帝之城（来世）和罪恶之城（现世）结合起来了，他认为虽然上帝已经预先设定了人死后的场景，但是没人有可以知道上帝选了哪些人（奥古斯丁-三位一体，上帝是永恒的，超越时间，至善全知全能），也就是说上帝是超验的超越人逻辑可以理解的，那作为个体在俗世里，又无法无能揣摩上意，你怎么知道你是不是上帝选中的了（勒布朗詹姆斯 - The chosen one)？加尔文认为虽然不知道，但是被上帝选中的人肯定能给上帝带来荣耀(honor)，这里可以参考下帕斯卡的赌注与上帝存在的证明。也就是说假设你是被上帝真的选中了，那么你就代表了上帝的荣耀，那你必须证明带来荣耀（加光），而不是给上帝丢脸。而在俗世这个光荣耀没办法比较啊，但是金钱财富、名声、地位却都是实实在在可以衡量的；你说你一个乞丐，对着亿万富翁说，我觉得我是上帝选中的了，你不是，他人恐怕也很难承认；上帝是超验的，不能用逻辑推理，但是作为具有社会性的人在俗世确是可以设定逻辑和考察标准的。所以新教的教徒从此可以在俗世里放开手脚大胆奋斗努力追求财富等等来突显上帝在自己身上展示的荣耀之光。

1700年初躲避英国宗教迫害带领大批清教徒移民来到美国东海岸开辟新英格兰地区的领袖约翰·温思罗普引用《马太福音》的一段5:14话将新殖民新世界比喻为“山巅之城”，这段话是：

“你们是世上的光。城立在山上，是不能隐藏的”

呼吁建立起一个基督教徒的榜样的社会来彰显上帝对他们青睐，展示上帝透过他们散发的光芒，表示他们信奉的教派才是掌握真理的“纯正”的上帝选中的教派-正宗。(另一得说就是打击异端，打压其他教派，包括处理其他教派的顽固分子，使用非常手段打击对手，也是合服上帝心意的，只要最后可以增加上帝荣光，比如1660年贵格会的玛丽·巴雷特·戴尔被清教徒绞死在波士顿，和后来的自称根据《摩门经》是耶稣基督最纯正的教会的犹他州盐湖城的摩门教等等太多教派纷争迫害了)。

1843年卡尔马克思的《黑格尔法哲学批判》导言中提到：

的确，路德战胜了信神的奴役制，只是因为他用信仰的奴隶制代替了它。他破除了对权威的信仰，却恢复了信仰的权威。他把僧侣变成了俗人，但又把俗人变成了僧侣。他把人从外在宗教中解放出来，但又把宗教变成了人的内在世界。他把肉体从锁链中解放出来，但又给人的心灵套上了锁链。

早期的资本主义有通过残酷的原始积累获取巨大的财富和极大的政治权利和地位，财富是上帝赐予人的礼物，特别是勤劳的工作者的礼物，自身也保持着节俭来积累更多的财富。同时他们也将大量的财富用于建造学校、教堂和社会福利慈善机构等等（比如黑人运动里被推倒的爱德华·科尔斯顿的雕像的英国布里斯托，爱德华·科尔斯顿就是通过奴隶贸易发家的，后面大量的捐助了布里斯托市的公共设施-建教堂、修路建桥等等慈善举动，布里斯托有大量的以科尔顿命名的设施），获取了巨大的名声，同时要求教徒过着节制的生活清规戒律（克己制欲望-比如富兰克林和克伦威尔），目的就是为了凸显上帝通过他们展示出来的荣耀的光，来证明他们正是上帝的选民。

但是随着科技的发展和去愚昧去神秘主义化，宗教的影响力也是在下降，人们越来越世俗化，越来越不太相信这套东西了；教徒们克己制欲不断累积的财富，财富通过投资越来越多，当财富到了一定程度，除了慈善等开销来获取名声，始终还是要消费出去的，慢慢的从制欲到了纵欲，人的本性开始难以抵挡糖衣炮弹（压制的越厉害，爆发的也越厉害了），钱开始从手段变成了目的，一定程度上减少人们对于上帝的依赖，因为老话说有钱能使鬼推磨，宗教的虔诚度也就没那么高了。新教中其中一派循道宗Methodism(循道宗虽不是穷人的宗教，但是是为穷人服务的)的创始人卫斯理John Wesley就观察到了财富积累之后的情形：凡是财富增加之处，其宗教精髓即同比例的减少。他在1786年的《Thoughts upon Methodism》提到了基督徒的财富观：尽你所能的赚钱，尽你所的节省，也你所的给予捐助。

“… What way, then, (I ask again) can we take, that our money may not sink us to the nethermost hell? There is one way, and there is no other under heaven. If those who ‘gain all they can,’ and ‘save all they can,’ will likewise ‘give all they can;’ then, the more they gain, the more they will grow in grace, and the more treasure they will lay up in heaven.”

卫斯理认为是尽自己能力范围找正道规矩挣钱，节省不是抠门不是吝啬，而是不搞铺张浪费攀比纵欲消费（极简生活），最后的给予你的一切指上面你挣钱节省最终都是将你所有的一切给予了上帝，上帝是人格神，是love/benevolent是仁爱,所以这些钱怎么给上帝了，还就是增加上帝荣光呗，比如上帝其他贫困潦倒的、弱势的、贫苦的子民和构建更紧密更公正更有温情compassion的社会团体呗。

卫斯理如何总结他对金钱的看法了？：

“(Money) is an excellent gift of God, answering the noblest ends. In the hands of his children, it is food for the hungry, drink for the thirsty, raiment for the naked. It gives to the traveler and the stranger where to lay his head. By it we may supply the place of a husband to the widow, and of a father for the fatherless; we may be a defense for the oppressed, a means of health to the sick, of ease to them that are in pain. It may be as eyes to the blind, as feet to the lame; yea, a lifter up from the gates of death.”

相比富人，宗教在穷人劳动人民更受追捧，富人们在经济基础优势上的理性、更好的教育和财富带来的独立性(清教徒巴克斯特Baxter看来财富带来的享受、怠惰和肉欲使人离弃了神圣生活的追求)反倒让他们更容易偏离宗教。托尔斯泰怎么对高尔基说的？：少数人需要一个上帝，因为他们除了上帝以外什么东西都有了；多数人也需要一个上帝，因为他们什么东西都没有。在观察德国早期资本主义社会里很多非常悲惨的工人待遇之后(法国里昂工人起义、德国西里西亚纺织工人起义），马克思在1843年的那篇文章《黑格尔法哲学批判》导言提到：

宗教里的苦难既是现实的苦难的表现，又是对这种现实的苦难的抗议。宗教是被压迫生灵的叹息，是无情世界的感情，正像它是没有精神的制度的精神一样。宗教是人民的鸦片。废除作为人民幻想的幸福的宗教，也就是要求实现人民的现实的幸福。要求抛开关于自己处境的幻想，也就是要求抛开那需要幻想的处境。因此对宗教的批判就是对苦难世界－宗教是它们的灵光圈－的批判的胚胎。

富人的痛苦可以很多渠道纾解，有的选；但是穷人的痛苦就只能求助于宗教的安抚-当然俗世里罗曼罗兰写给“世界简直透不过气来”的苦难时代的英雄传记或许能到来更强更有力的呼吸和更纯净空气来心灵污迹，但是这些痛苦世界里英雄难以接近远不如上帝那么亲近而且全能（某种程度上可以解释美国社会的分裂和反智反理性现象、韦伯后来转变为了理性的悲观主义者-理性的美梦到了梦魇的转变)。

随着理性科学的发展，上帝也从人格神变成了自然神，到了尼采，进一步说的：上帝已死。上帝带来的这个道德体系已经瓦解了，外部所依仗的终极价值衡量标准没了，人可以跳脱出来成为自己的主人，有了更多的自由但是也就需要为自己的行为负责，自己决定自己的一切，只有强者才能这样，因为弱者才会依附于外在把自己交出去给他人（没有上帝了），从此人成为了萨特说的被判了"自由的无期徒刑”。问题是少数人掌握更多的财富、更好的科学理性教育，上帝也就越加没有吸引力，自然而然的都想成为超人（意志的强者），主宰自己的命运，为什么自己的生命不能永恒了，即使不能永恒怎么确保我的后代可以保持或者扩大这种优势，成为超人之超人了？

可以看到近些年来有硅谷大佬了比如Peter Thiel往身体里注入年轻人的血液的尝试、塞吉·法盖特（Serge Faguet）“生物黑客技术”（炼丹?)等等，富人们将更多钱投入到了永生的科技研发之中The Rich Will Outlive Us All Tech billionaires’ attempts to beat death will not democratize longevity，硅谷和生物科技领域联手的兴起，尝试打造”不死之身“；著名的传记作者Walter Isaacson最近将要发布的一本书《The Code Breaker: Jennifer Doudna, Gene Editing, and the Future of the Human Race 》，他在WSJ发表的一篇预热文章《What Gene Editing Can Do for Humankind： The new biotechnology called CRISPR offers the prospect of defeating lethal viruses and curing genetic diseases, even as it raises serious moral questions》提到了通过人类胚胎的基因编辑可以避免一些先天性的疾病比如镰形细胞贫血症（Sickle Cell Anemia），有希望可以大大减轻人类的痛苦。但是这样一来，也有人担心这些被“设计”的婴儿可以让富人相对于穷人有了遗传的优势，不仅有财富地位的，将来可能还有身体机能上的。那么极有可能富人相对穷人除了后天在系统结构组织上的优势，可能还有先天上生物优势。

[2020年关于现代经济的一些思考(五)]

2020年关于现代经济的一些思考(三)

2021-02-03T14:55:32+00:00

1980里根经济学-新自由主义

自从尼克松之后的福特和卡特两节总统内，高通胀高失业率和经济停滞一直是一个问题。有意思的是现在全世界都陷入了通缩的节奏，那个时候通胀确实给美国人带来了非常痛苦的回忆，两相对比令人唏嘘。为了控制通货膨胀，他两都实验了自愿的价格薪资控制指导Voluntary Wwage-Price Guidelines，不是尼克松的必须的强制的mandate，只是提供了建议和指导,看起来有进步。福特还搞了一个公众运动叫”鞭笞通胀Whip Inflation Now!“

这也有点幽默的，但是他两的措施都没有效果，相反更像是通胀鞭打他们的屁股。美联储这边，一直以来没有办法完成他的经济目标，也造成了一定程度的信任危机。

在Recession of 1981–82沃尔克上任初提到的他的目标是降低通胀：

In terms of economic stability in the future, [inflation] is what is likely to give us the most problems and create the biggest recession” (FOMC transcript 1979, 16). He also believed that the Fed faced a credibility problem when it came to keeping inflation in check. During the previous decade, the Fed had demonstrated that it did not place much emphasis on maintaining low inflation, and public expectation of such continued behavior would make it increasingly difficult for the Fed to bring inflation down. “[F]ailure to carry through now in the fight on inflation will only make any subsequent effort more difficult,” he remarked (Volcker 1981b).

1980年通胀率大概10%左右，保罗·沃尔克做了其他不敢做的，开启了紧缩的货币政策，大幅度提高利率到20%。这样大幅度短期提高利率，一定是会影响短期的经济衰退的，也就是一般来说政客是不会做的，因为这样会短期得罪公众让大家过苦日子，尽管你认为长期对他们而言是更有利的。1981-1982年经济衰退甚至是大萧条之后最严重的经济下行，1982年失业率达到了11%，甚至高于09的金融危机。一度在经济情况太差，连国会施加压力质询他希望他放松货币政策来缓解经济形势，但他仍坚持如果不够通胀率降下来就带来长远更严重的灾难。这些措施在1982年得到了好转，通货膨胀降到了5%，利率开始下跌，失业率也开始下跌，到了1987年他卸任膨胀率大概是3.4%。沃尔克激进的降低通胀的决心使得社会上有了一个稳定预期 - 未来通货膨胀率一定是慢慢下来，这种稳定可靠的预期也有利于商业活动的进行. 而这个要给后续美联储的政策制定等等带来巨大的影响。

也有人质疑沃尔克的紧缩货币政策是不是一定要么严苛？短时间的巨大震荡Volcker Shock，为了控制膨胀率，让无数的人失业，关闭那么多工厂和矿工。农民，工匠，建筑工人纷纷写信给沃尔克抱怨和抗议。同时美联储的升息使得美元回流美国，造成拉美的债务危机。

扯远点：沃尔克跟他的身高一样成为了美联储历史上非常伟岸的形象，颇有马基雅维利《君主论》和边沁功利主义(有兴趣看看Michael J. Sandel(麦克尔·桑德尔）的公开课《正义》）的结合。马基雅维利（1469—1527）的时代14-15世纪也是内部四分五裂的城邦国家内外交困的时代，夹在法国、阿拉贡等国家中间（乱世），期待羸弱的洛伦佐成为果敢的具有virtú的的强人，领导至少佛罗伦萨实现统一应对威胁。他详细描述了君主、将要成为君主的年轻贵族（美第奇家族有两位是教皇）都应该怎么做具有什么样的气质才可以达到上面讲的他理想中的伟大君主。其中有一点是他认为君主应该代表了千万人的利益（重），决不能以个人的观点视角来要求自己（轻），对于评论名声的不要太看重，短期不利大众的利益可能会带来非议但是长期来说对于大众是有利（公众福利）的话，他应该坚持自己的看法；

马基雅维利描写的如何在内外交困下面君主如何怎么样才能快速统一，新国家如何先站住脚跟的问题，这个就跟俾斯麦时期的德意志联邦很像了，内部四分五裂，北有新教南有天主，同时1806年被拿破仑的法国打败，拿破仑开进普鲁士的勃兰登堡门将其上面胜利女神雕像拆下作为战利品带回巴黎(有意思的是1871年在俾斯麦击败了拿破仑的侄子拿破仑三世，德皇在法国的凡尔赛加冕），俾斯麦应该就是马基雅维利笔下最佳的君主的形象了；俾斯麦普鲁士打奥地利时，初期在“军队碰巧有了一个国家”的普鲁士面前奥地利一败涂地，国王和民众群情高涨，直呼要打到维也纳，此时俾斯麦力排众议强烈坚持自己的当初的看法（快速给奥地利一个打击，目的在于给南方这些德意志邦看看他们依仗的盟主多少弱鸡，同时避免战争升级导致法国的介入），甚至俾斯麦自传里的说法他以辞职来要挟退兵甚至是急哭了，最终说服了威廉一世阻止了战争的继续和升级，保障了他长远的政治目的（德意志的统一）；俾斯麦在自由和保守派中间进进出出，并不拘泥于某种形态，同时他对于人的仇恨心理有很深的研究（普法战争的电报），最后他本人是相当理性主义的人，实施了社会福利的政策，改善了社会的矛盾（不过俾斯麦认为的是国家发福利就是为了让那些群众满意，不然如果发生革命的话，就要花更多的钱来摆平）。

俾斯麦认为在同一德意志过程对于那些小的邦国应该无情碾压过去，因为他们的声音太小，也没人会听见也没人会理会；某种程度上保罗·沃尔克也是沿袭了下来，甚至可以怀疑他是不是跟拿破仑一样把君主论作为枕边书。

另说就是普遍把马基雅维利主义认为是讲究不择手段来获取个人利益的学说，被称为厚黑学，这个观点还是有失偏颇的。结合他的时代背景，马基雅维利深刻洞见了政治和道德的关系，特别是在非常时期，在未形成稳定的国家或者秩序之前，需要采取这些非常规的，甚至非道德残忍的手段，牺牲小部分来成就大部分人的利益，这个事情在资治通鉴里屡屡发生，无可否认他把人民百姓给简单化抽象化了。但他本人还是认为这些应该是短期的手段，而新君主在巩固好政权之后就应该： “使用那些建立在你自己的行动和美德之上的方法才是“唯一正确和持久的方法。”

这里的关键在于否认为使一些人享受较大利益而剥夺另一些人的自由是正当的，这也正是密尔在边沁的功利主义上面延伸引入个人的权利的容纳的考量。价值问题不是科学问题，不能单一推导决定，而只能是解释，解释有不同的视角深浅，而价值依存的环境也会变化，正是这种非理性和动态性，所以用二元黑白的方式来评判，是非常狭窄的。

而马基雅维利主义被历史各种政治野心家（克伦威尔，普鲁士腓特烈大帝，路易十四，拿破仑，希特勒，墨索里尼）用来有心得解读来尝试正名化自己的非常规行为，这个不能抹杀马基雅维利的意义，他提供了一种洞察世界的视角，让我们能更好的理解行为现象背后的逻辑。

1513年，从高级公务员到被投入监狱，被诬告针对美第奇家族叛乱，收到酷刑，晚年的他回到了乡下做了大量的阅读研究开始着手写《君主论》

当夜晚来临，我回到家，进入书房。在门槛处，我脱掉沾满泥土和秽物的工作服，穿上驻外使节的正装（因为他曾经做过驻外使节），端正仪容之后我进入往圣先贤的殿堂(书籍). 在那里，我受到热烈欢迎，然后我开始品尝那我觉得有养分的唯一食粮，如饥似渴地细嚼慢咽。我毫不腼腆地与他们交谈，让他们解释他们的行为，而他们总是不厌其烦的回答我。四个小时无忧无虑的时光就这样匆匆而过，我忘掉了所有的烦心事。我不再恐惧贫困或死亡。我的生命完全穿越了他们。 - Machiavelli，A letter to Francesco Vettori

他对于读书的热情和获取知识的热情也体现了他自己的一致反复提到的virtú。正如《Trump’s lawyer plays a video of Democrats and celebrities advocating violence at impeachment trial》说的：

“The election of Donald Trump did not create the left’s hate….it revealed it.”

特朗普的当选并没有创造或者造成左翼民主党人的仇恨，他只是展示了揭露了这些仇恨，问题不只在特朗普，在于这现象背后的社会问题和矛盾；没有特朗普，也会有其他人作为这个代表，这才是需要值得正视的。

回到正题，但这不妨碍本伯南克将他视为代表了美联储的独立精神，他诠释了如何做政治上不受欢迎但是经济上必须要做的典范

“came to represent independence. He personified the idea of doing something politically unpopular but economically necessary.”

本伯南克后面还吹嘘了沃尔克的继任者-格林斯潘，大赞因为沃尔克带来的美联储的独立，是的格林斯潘放开手可以追求宏观经济的稳定，经济增长同时经济波动的幅度很小- 称赞格林斯潘治下的时期是大稳健时代Great Moderation。

1995年沃尔克后续加入了华尔接的投资公司成为主席，有意思的是在09金融危机时，沃尔克提出了Volcker Rule:限制商业银行用自己的钱在其他股票债券和其他衍生品上赌博。这个也成为了2010年的《多德弗兰克法案》中其中一条法令。有意思的是：1979年沃尔克刚上任就bailout保住过了克莱斯勒汽车，结果08年克莱斯勒又被bailout一次，连着两次被bailout这样光荣记录应该前无古人后无来者吧 - 但是bailout很明显不是弗里德曼喜欢的自由竞争。

正如俾斯麦的成功强势离不开威廉一世的支持和容忍，沃尔克的政策也离不开里根的支持。

里根经济学

特朗普喜欢用里根的参选标语（Let’s make america great again），同时特朗普是一个政治素人是企业家娱乐家，而里根曾经是电影演员(主要是牛仔片）。但是不同的是里根参选之前是有担任过八年加州的最大州的州长，有从政的经验，而且跟演员时代也隔开了很久了。里根参选时候是有70岁，是当时年纪最大的总统参选人，他的从影经历仍然让大众对他的领导能力产生了疑问。1980s年的英国政治讽刺木偶节目Spitting Image中《Ronald Reagan on Spitting Image - the original US president without a Brain!》里面调侃里根记不起来观众的提问是什么，意在表示对于里根能否处理伊朗人质危机等等有疑问（真正可虚可叹的是里根90年之后确实的了老年痴呆症）。

里根在1981年的就职演说C-SPAN: President Reagan 1981 Inaugural Address里提到:

“It is no coincidence that our present troubles parallel and are proportionate to the intervention and intrusion in our lives that result from unnecessary and excessive growth of government.”

核心： ”Government is not the solution to our problem. Government is the problem.“

里根提倡共和党的理想中的小政府，缩减政府的开支，减税和减少过度的政府管控（比如之前价格工资管控）。他相信的是一个自由市场和资本经济可以解决目前这个国家的问题。这个放任经济跟货币主义提出的经济的自由来自政治的自由不媒而合。

但是他一上台就是1981-1982年仅次于大萧条的经济萧条，他激进的将个人所得税中的高档所得税率从70%降到50%，企业的税率从46%砍到了34%，承诺减少政府开支和放开商业的管控，配合美联储控制通胀。

到里根下台时，政府开支比卡特时期相对来说减少了，而且赤子在他任期内虽然没有缩减但是增长程度不大（尽管星球大战国防开支激增），高累进税率从70%降到28%，资本利得税率从28%降到20%，企业所得税率则从46%下调到33%，放开了价格管控，美国经济恢复了增长。他在告别演说《President Ronald Reagan’s Farewell Address to the Nation. January 11, 1989》回顾了自己的总统任期，主要讲了一个恢复了美国经济的增长，一个恢复了美国人民的士气。顺带提到了一个故事：1981年他刚上任参加一个在加大拿大举行的经济峰会，开幕式是7个工业国家的首领的一起参加的正式晚餐，里根坐在那里像是一个刚到学校的新学生，晚餐的对话听到的都是法国人或者德国人的名字，他们互相告诉自己的身份并用名字互相称呼（显得互相紧密），某个时刻里根唯唯诺诺地靠过去说：”你好，我是Ron小罗”(没有说Ronald罗纳德)。从那年开始里根开始恢复经济的行动-减税，松管，减少支出等等。两年后在另外一个经济峰会（有着类似的成员），在盛大的开幕式上突然有一刻里根看到所有人坐在那里都望着自己。然后其中一个打破沉默说请您告诉一下我们美国的经济奇迹么。(虽然总统离任演讲都是吹嘘自己的功绩，但是这个还能看出来一些的)

关于减税的问题，里根认为削减税收，当然政府的收入下降，但是因为减税企业能得到额外的钱可以用于投资和雇人扩大再生产，那么如果基数足够大的话，新增的这些产品和雇员所带来的税收可以弥补减税的那部分。这个减税的幅度和收益的关系就可以通过拉弗曲线(Laffer Curve)来得到：

一部分观点是一开始税率太高，大幅度减税确实可以达到这个双赢，但是一旦税收到了后面在使用这个策略结果效果就一定好了。盲目减税不一定能带来好的效果。

这个也被称为涓滴经济学Trickle-Down Economics ，通过减税-高级税也就是富人税(Vox - How tax brackets actually work, 美国按照收入不同分等级，指的是高档收入人和大企业，大投资人大银行等等的税收）可以让企业招人扩大生产，银行可以放贷更多钱，投资者可以买更多企业的股票或者公司，所有这些最后都会实惠到底层的工人。这些工人也会花费他们的工资来驱动需求的增加和经济的增长。但是这个理论和制度也都是作用范围的，里根时期70%的税收确实太高，所以减税的驱动效果很好；但是一开始有效果并不代表一直用都会好，超过某个平衡点时，可能带来的不是好处更多是坏处了 - 企业家开始衡量拿钱投入生产的回报跟其他途径的回报相比哪个更合算。从里根到小布什到特朗普，随着税率越来越低，我们看到减税的效果对经济增长的助力作用明显减弱。

里根开启的金融管制的放松，包括放开储蓄银行的低利息限制和存款比列的限制，进一步资产证券化-其预先的设想是让更多美国人容易付得起获得房屋。这些储蓄银行因为利息太低无法吸引客户，只能是拿着储户的钱偷偷去投资高风险高收益的金融产品。资产证券化，比如住房抵押贷款支持证券（MBS）这一时期得到极大的发展。大概原理如下图：购房人找银行借款30年100万，这里面假设收取的利息每年加起来总共是10万块，也就是银行这100W块30年的总的回报是10W，需要等待很多年才能把钱收回来，这中间没办法可以都放贷给其他人。如果这家银行只有100W，那么只能给一个贷款，也就是只要一个人才能买房。但是如果银行说，这样吧我这个打包一下给第三方金融机构，把购房人的每期的利息本金等等都给第三方，第三方在把这个分配打包证券等等卖给需要的投资者。银行让利比如10万回报中给5万的给第三方，第三方将这5W中4W作为利息给到投资者，这样一来银行快速收回了钱还有回报（100W+5W），它可以放贷给另外一个需要的人；第三方的投资者拿到了5W的投资回报；购房人也拥有房屋 - 一举三得。从这来看这个金融创新最开始确实很好，但是为什么到了08年却成为了大问题了？监管松懈，过渡金融化创新。

里根的金融松管到了克林顿这里进一步扩大，克林顿认为这些纷繁的监督管控条目限制了金融的发展和创新，比如我们之前提到的1933年的银行法案《格拉斯-斯蒂格尔法案（Glass-Steagall Act）》中分离商业储蓄银行和投资银行业务，目的是限制投机。从1980年之后金融银行业一直游说政府废除《格拉斯—斯蒂格尔法》，始终未成功，直到1999年比尔克林顿签署法案废除了该法案Gramm–Leach–Bliley Act，这样一来商业银行就可以同时作为投资保险银行，银行兼并的寡头时代来临，也促进了08年金融危机的发生。花旗银行作为储蓄商业银行可以用储户的存款来投资证券化的资产博弈-MBS抵押贷款证券和债务担保证券CDO。一年之后2000年 J.P. Morgan & Co和Chase Manhattan Corporation合并成为了JP Morgan Chase摩根大通。

在1999年签署Gramm-Leach-Bliley Act时候，从视频中可以看出Signing of the Gramm-Leach-Bliley Act克林顿提到：

“This is a day we can celebrate as an American day” and that ”the Glass-Steagall law is no longer appropriate for the economy in which we live” and “today what we are doing is modernizing the financial services industry, tearing down these antiquated laws and granting banks significant new authority” and “This is a very good day for the United States.”

成年人不能穿孩子时代的衣服，为了现代化银行加强竞争力，我们需要赋予银行更多的权利。

商业储蓄银行吸收存户存款，而且存款是有联邦存款保险公司（Federal Deposit Insurance Corporation）保证的（上面1932年罗斯福的储蓄保险政策）所以他们想要那这些被政府保险的低成本存款来跟投资银行竞争。你想想即使储蓄银行拿这些钱去投资上杠杆高风险高收益，如果成功，挣得利润归自己；如果失败，有政府的保险，如果储户找上来要钱，但是银行亏空了，反正有政府兜底。这个正是《格拉斯—斯蒂格尔法》想要避免的。

到了08年，其他的转型成为了双行（即是商业也是投资），其中的巨头有花旗，摩根 Chase和美国银行，只剩下有5家出名的专业投行。金融危机中，其中 Lehman Brothers雷曼兄弟破产，Bear Stearn倒闭被JP Morgan Chase收购，一家Merrill Lynch存活但是不得不被合并到美国银行，第四和第五家高盛Goldman Sach和摩根斯塔拉(Morgan Stanley)也是非常危险，办法是：一夜之间他们宣称他们已经有单纯的投行转成了既有商业也有投资的联合银行，所以应该符合被FDIC联邦存款保险公司保险救济和美国政府的隐形支持的的名额。

2015年比尔克林顿在这篇采访中被问到关于为什么从对加强监管只是开空头支票，他如何解释universal bank(双行合并）跟后来的危机关系：

I’ll tell you exactly why. I think that politicians–particularly now, in the aftermath of this crash–fear that anything they do will be held against them later if anything bad happens. Look at all the grief I got for signing the bill that ended Glass-Steagall. There’s not a single, solitary example that it had anything to do with the financial crash. And in fact, a study done afterward said that the unified banks were actually slightly less likely to fail than either the commercial banks that overloaded on subprime mortgages, or the investment banks, like Bear Stearns, Lehman Brothers, and others. Given the polarized nature of our politics, I think they’re afraid that if they do this, which they ought to, the counterpressure will be too great. And I think that’s a big mistake. You just can’t be afraid to be second-guessed. If you’re in politics, it’s part of your business to be second-guessed.

克林度认为他的取消管控的措施，包括加速银行业的兼并和复杂的金融体系，跟08年金融危机的发生没有一丝丝联系(WTF怎么听起来有点像I did not have sexual relations with that woman, These allegations are false!？)，相反他认为联合银行反而不容易倒闭。

而正是克林顿开启了金融化时代Age of Financialization.

2008年经融危机

2000年的美股的Dot-com bubble互联网泡沫，一方面是投机互联网新技术（任何带有.com结尾都会被疯狂追捧），一方面美联储的低利息，还有就是自由化后监管缺失；2000年泡沫的代表就是宠物网Pets.com,做网上宠物用品销售的，消费者导向”,“砸钱营销A medley of Pets.com TV Commercials，资不抵债，最后破产；2001年的财务造假而破产的安然事件严重打击了商业和投资者的信心；

ps: 上面提到的Jeremy Grantham在2021年说有一个超级大泡泡A Historic Market Bubble里提到了他之前的2000年的dot.com泡泡的经历(类比1929年华尔街股市大崩盘前肯尼迪的父亲去擦皮鞋的时候，鞋童一遍擦鞋一遍对股票侃侃而谈，于是他回去想了想不对劲赶紧卖出了），里面就提到了Pets.com：

“My favorite example from 2000 with lunchtime greasy spoons that we went into in Downtown Boston, and they all had quite a few television sets and they were all playing replays of the Celtics and the Pats. For a few months there in late ‘99, early 2000, they’d closed the sports ones and trends for them to CNBC. And we’d have talking heads talking about the latest pet.com’s. That was terrific, that gave you just a month or two to plan accordingly, plan evasive action. And I would say we have passed the acceleration test brilliantly, and I would say we have passed the crazy behavior test brilliantly. And a lot of the craziest behavior actually taken place this year in January, February.”

回到正题，在互联网泡沫和911之后美联储不得不降息来保持经济的刺激，小布什宣布自1981年之后最大规模的减税方案，这些多出来的钱希望找到回报利率更高的投资项目，一方面国际方面经过之前的亚洲金融风暴之后新兴国家的收益不太行，所以美元也都回流到了美国。这些钱都流到哪里去了？房地产市场。

来自达里奥的债务危机：

2004到2006年之间，房屋价格上涨了30%，并且相比2000年上涨了80%。假设设一个家庭用5万美元支付了一个价值25万美元房子的首付，然后房价从25万上涨到了35万，这个家庭的投资翻倍。这也带来更高的借贷比例，更多的人进入房地产市场。随着买房带来的财富效应，越来越多的人开始将房地产进行再融资，增加杠杆来投机。

但是我们贷款买房这种银行需要审查贷款人的条件的，比如这个人的收入水平，拥有的资产，否则还不起贷款就麻烦了。但是这些能带的起款的优质客户相对来说太少了，如果房价一直上涨的话（从2000年到2006年一直都是），那么其实这个人是否还的起利息有没有条件似乎就不是那么重要了。本来这些非优质客户的贷款（次级贷），一开始是这些银行才会玩的，这样堵高风险但是回报率更高，并且当房价看起来一直再涨，这些银行会越加大胆的上杠杆；这样一来这些政府支持GSE的共用性的放贷机构比如两房- 房利美和房地美，为了跟他们竞争，也不得不跟上，开始接受更多的次级贷。

为了规避金融监管，银行还会利用金融创新影子银行ShadowBanks在银行之外吸收更多的人放款，因为这些不在银行监管之内，那么这些获取贷款的人条件更送，进一步加剧了次级贷的泛滥。美联储盯着通货膨胀率和结业率两个目标，但是这种情况下跟1989年日本是一样的，居民消费价格水平并没有升高，因为钱都流入了资产房地产里，这样一来美联储并没有察觉太多通胀的（有轻微的因为能源输入性涨价的通胀但是问题不大），也没觉得经济有什么问题。

2005年10月20号，当时还不是美联储主席的伯南克写给小布什政府的经济展望THE ECONOMIC OUTLOOK中主要是评估8月份发生了hurricane katrina飓风卡特里娜(可能我只记得麦迪和阿里纳斯在慈善救助比赛里互飙三分的场景）给经济带来的影响和当前未来的经济展望。关于房价上涨是这么说的：

House prices have risen by nearly 25 percent over the past two years. Although speculative activity has increased in some areas, at a national level these price increases largely reflect strong economic fundamentals, including robust growth in jobs and incomes, low mortgage rates, steady rates of household formation, and factors that limit the expansion of housing supply in some areas. House prices are unlikely to continue rising at current rates. However, as reflected in many private-sector forecasts such as the Blue Chip forecast mentioned earlier, a moderate cooling in the housing market, should one occur, would not be inconsistent with the economy continuing to grow at or near its potential next year.

他觉得尽管可有房价的投机行为，但是全国而言，房价的过去两年上涨25%主要反应了强劲的基础面，包括就业和薪资的强劲提高，低房贷利率，稳定住户形成率和有些地区限制房子供给的因素。房地产价格不太可能会以如此幅度继续上涨。但是正如很多私人行业预测的，如果房地产市场出现适度降温，那将不会与明年经济继续以明年或接近其潜在潜力相矛盾（即使房价降温，明天经济也可以持续增长）。

讽刺的是当伯南克准备上台的06年，其实05年底时（这篇报告时），次贷率已经几乎达到了顶点。推动泡沫形成的，一定不仅仅是低利率水平，而是还有更容易的信贷，更宽松的监管，更加危险的金融创新。

其中如何吸引更多人购买了？哪怕是没钱的人？实际上在房价一直上涨的期间，他们如果他们家的宠物可以贷款的话，我相信他们都会给贷。来一张他们如何吸引购房者又可以让购房者不必担心资质的问题的照片。

我们可以看到1.无需文件的贷款(no-document，你不用提供一堆工作和收入证明的文档)；2.一开始的利息很低1% - 实际为可调利率抵押贷款ARMs; 3.批准率很高,4/5通过率；4.没有提前的花费（首付）; 5.“还息不还本” Interest only loan ; 6.「不查收入（stated income) - 收入你自己填

这些都是短期很迷惑性的，比如(ARMs)可调利率抵押贷款的，一开始几年利息很低，慢慢的利息就会变高；还息不还本” Interest only loan都得要求一定时间内归还的；特别当中介担保房价涨涨，肯定没有风险时，你也可以获利的时候，贪婪就容易让理性迷失。就跟我们现在泛滥的金融网贷，白条360花呗，听起来说的都是日还一两块还不到一瓶水的价格，好像这个钱放在那里是白来的，非常就有迷惑性，但是价格计算非常复杂（日利率和等额还息）这些隐藏的套路和协议，是借钱者（特别是迫切借钱的人）在感性之下（或者是冲动消费）无法考虑到的。

这些人的钱是拿到了，但是投资者也不是傻子，这些钱是有风险的。怎么处理这个矛盾了？这里面有很多非常复杂的术语和套路，简单的说是以钱生钱，放大杠杆，跟之前马云的蚂蚁金融的差不多，空转套利，很少一部分才会流入实体当中。我推荐看看这个视频The Crisis of Credit Visualized - HD和李永乐的《蚂蚁金服如何把30亿变成3000亿？资产证券化如何导致美国次贷危机？》

图片来源：http://m.wjw.cn/baoxian/news-742504.html

当房价上涨的时候，买房人，银行和贷款机构，投行，对冲基金和评级结构都可以获利；但是当房价下跌的时候，就会进入下降的恶性循环。这些投资银行创造的大量金融创新产品CDS，CMO，将高风险通过巧妙而复杂的包装之后，实际上没有人能确切的可以判断这个资产或证券到底风险是怎么样的（透明->不透明），实际上只要挣钱没人会关心，从而带来了巨大的风险。

美联储从 06年开始意识到了次贷的问题，但是没有想到是全国性的问题，但是已经开始升息来一直经济过热。2007年三月为了应对慢慢升级的房地产市场中的不良贷款问题，在《Manageable Threats Seen by Fed Chief》中伯南克提到他不认为逐渐上涨的风险借款人的违约率（这些有不良贷款记录的人）会威胁到整个国家的经济活力。《2007年3月28日 - The economic outlook》在这个经济展望里他似乎更在乎是通胀还是通缩的问题，关于次贷问题：

At this juncture, however, the impact on the broader economy and financial markets of the problems in the subprime market seems likely to be contained.

他认为这个次贷问题在这个时刻似乎在全局是被控制了。但是他也觉得有可能未来情况变糟糕：

Overall, the economy appears likely to continue to expand at a moderate pace over coming quarters.This forecast is subject to a number of risks. To the downside, the correction in the housing market could turn out to be more severe than we currently expect, perhaps exacerbated by problems in the subprime sector.

但是美国股市听了这话，在4月到5月继续上涨，创了历史新高。

就雷曼兄弟来说，150多年历史的雷曼兄弟一直扮演着中间人角色（借钱方企业和出钱投资方个人联系起来）一直在每个时代都正确的押宝，1850-60年的棉花，世界末转向金融新生的高风险企业比如零售retail（Sears, Woolworth, Macys)，然后一直转向新的行业比如电影(Paramount派拉蒙 &Fox福克斯),然后是航空业(环球航空公司twa),然后消费品和汽车（福特Ford)，技术(intel因特尔)雷曼兄弟几乎都是在行业刚起步风险高的时候正确的时间进入，推动了行业的快速发展，自己也一步步壮大，成为一家非常出名的投资银行。2000年之后然后雷曼兄弟大规模押宝房贷-次级贷，上涨的房价使得他们的利润不断上身，以至于他们在06-07的利润达到了高峰，07年利润高达4.5亿（几乎是五年前的4.5倍），不过反过来进一步强化他们押宝次级贷，低利率低成本的借钱而后不断上杠杆。

但是利率上升，自然短期利率的调整会有严重的影响，比如可调利率抵押贷款和还息不还本，会加重还不起贷款的情况，同时利率传导到长期的按揭利率(这个需要时间，但是期望预期已经成立)，让资金从长周期的资产投资中抽出来，导致资产价格的下跌，也就是房地产价格的下跌。还不起贷款，房价下跌，投行和保险都会跟着下跌，就像是恶性循环，直接引爆政整个金融市场。随后股市暴跌，金融市场奔溃，贝尔斯登可能挤兑要倒闭，其他花旗和美林面临着巨额的坏账，银行投行保险机构等等可能一家家被清算。

可以看看大空头里的Michael Burry在2011年的分析金融危机的的讲座视频《Missteps to Mayhem: Inside the Doomsday Machine with the Outsider who Predicted and Profited from America’s Financial Armageddon》.(ps: 虽然我不炒股投资，但是可以结合后面散户WSBvs华尔街空头在GME斗争事件的发起人Roaring Kitty一直以来怎么分析GME的数据找出空头如何过渡杠杆的并且结合价值投资等等100%+ short interest in GameStop stock (GME) – fundamental & technical deep value analysis可以看出来信心任何时候还是来自做好你自己的功课(do your homework)，不是简单的运气而已，不然如何在外界疾风中坚持自己的信念和信心(劲草)，不能是空气吧。)

最后只能政府出面出台了TARP坏账购买计划）来给市场注入流动性，通过给金融企业注入资金，逐步恢复资产价格，带来信心的提升，随后银行也可以进行再融资。高盛和摩根斯坦利也获得了美联储的保护（上面提到的）。

美联储没有救投行雷曼，雷曼倒闭，房利美和房贷没被纳入财政部监管并有财政部保证其债务，MerrillLynch被美国银行收购，AIG提供保险的接受了紧急流动性的支持，WashingTonMututalBank被关闭随后被摩根 Chase收购，同时摩根 chase还收购了破出售的贝尔斯登被。美联储一个是有毒资产的最后借款人，另一方面降低利率到0的水平和下调贴现率窗口，同时释放流动性。

这次仅次于1929大萧条(Great Depression)的08年金融危机(Great Recession)让伯南克碰到也是巧了。为什么了伯南克正好是一个喜欢研究1929年大萧条的人？应该是痴迷1929年大萧条的，就像按照他自己的话说是跟其他喜欢研究美国内战爱好者是一样的。这篇2005年的WSJ文章调到《Long Study of Great Depression Has Shaped Bernanke’s Views》相当长时间研究大萧条的历史已经塑造的伯南克的观点和看法。他觉得大萧条的那些教训在今天仍然有用，特别是其中通胀和通缩的问题。

The Depression, he contends, has taught the importance of avoiding both deflation – that is, generally falling prices – and inflation. It has also shown the threat that falling asset prices – such as, potentially, in housing – and weakened banks can pose. Most important, it shows the damage the Fed can do when it follows wrong-headed ideas.

文章里透着乐观祥和：

Today, growth is strong, unemployment is just 5% and housing prices are lofty. The Fed is raising interest rates to prevent a rerun of the 1970s, when high oil prices and a loose monetary policy unleashed double-digit inflation.

可以看出来美联储和伯南克更在意的就业率和通货膨胀，而房价还很坚挺。

Mr. Bernanke has argued that an even more important benefit would be to prevent the Fed from stumbling into deflation. Its inflation target would be a lower limit as well as a maximum.

解释了美联储一直将通货膨胀的目标设置为2%，不高不低，而且方便灵活使用各种工具，有调节steering的余地。

He says Mr. Bernanke’s interest-rate actions thus may resemble Mr. Greenspan’s maneuvers of the early 2000s: aggressive cuts when recession threatens, gradual increases when the economy recovers.

这点得到了印证，伯南克很快将利率达到了0，快准狠。

Perhaps the most important lesson Mr. Bernanke draws from the 1930s is the importance of thinking creatively when faced with unusual economic challenges. …Mr. Bernanke has written that President Franklin Roosevelt’s most important contribution to ending the Depression was "his willingness to be aggressive and to experiment."

伯南克认为在1930大萧条期间的最重要的教训是：当面对不平常的经济挑战时，需要创新的思考。他认为罗斯福之所有结束大萧条最重要的原因是：他愿意变成激进并且去实验。

Mr. Bernanke took particular interest in Japan’s struggle with stagnation and deflation. Like the Fed in the 1920s, the Bank of Japan set out, in the late 1980s, to prick a bubble, this time in stocks and property. Like the U.S., Japan then experienced deflation and depression, though far milder than the U.S. did. Like the Fed in the 1930s, the Bank of Japan, in Mr. Bernanke’s view, bore much of the blame.

伯南克还特别对日本的通货紧缩的失落的三十年特别感兴趣，经济滞涨，物价通缩，他认为日本央行应该承担很多造成经济衰退的责任。这段历史也特别有意思，因为伯南克在08年的非常规武器里很多都学习日本央行，比如QE量化宽松，都是日本央行先实践探索的（参考日本央行BOJ在2001年开始探索）。

2002年伯南克在面对经济恢复缓慢，持续下降的膨胀率（恐惧通货紧缩，大萧条的紧缩吓怕了美国人，1970年代的通货膨胀也吓坏了美国人）的情况时候，美联储看起来主要武器没有弹药了，基准利率已经很低了。如果此时提高利率，那么通缩会更严重。这个常规武器没效果了（Fed funds rate - principal instrument - fell to zero）。

He said his preferred alternative tool was for the Fed to reduce long-term interest rates by buying bonds, as it did in the 1940s. Long-term rates have a more direct impact on mortgage and corporate loan rates than the Fed funds rate. He also suggested more-exotic tactics such as a tax cut financed by printing money, or pushing the dollar down on world currency markets

他的建议是效仿1940s二战之前一样，通过直接购买去债券市场购买债券来降低长期利率。因为长期利率更能直接影响比如房贷和政府贷款，周期更长(后面会讲到，因为短期利率传导到长期是需要时间的)。甚至可以有更特别的比如印钱让美元贬值和印钱来支持减税等等（让人们手头上有更多的钱）。

总的来说他当时提出的这些主张是因为太大胆没有被接纳。

“Once you’ve made the commitment, how do you get out of it?” says Kenneth Kuttner, then an economist at the Federal Reserve Bank of New York and now at Oberlin College. If investors even suspected the target wouldn't last, they would unload bonds, driving long-term rates up.

这个很合理时，实际上08年伯南克创造性的解决了这个，就是收益率曲线控制/Yield Curve Control.到了08年，伯南克这个喜欢研究大萧条的爱好者真的碰到了经济危机，使用常规武器肯定是游刃有余的。

历史总是那么相似，在面临大萧条的时候，选择释放天量流动性并且压低利率价格，成了政府以及央行一致行动的决策。这一次美联储出台的是量化宽松。我们从下图看到，货币供应在1929年大萧条之后出现了巨大的增长，并且伴随着利率水平见底。而这一次同样出现了巨大的增长，同样伴随着利率水平的见底。

但是问题跟02年他碰到的问题差不多，美联储货币政策的常规武器打光了，但是对于恶化不见好转的经济形势还是没有啥太大效果，去哪里找速效救心丸了？

美联储的武器库

上面我们讲到了美联储的常规武器是通过公开市场操作Open Market Operations(OMO)主要是买卖政府债券影响短期基准利率利率，从而影响到货币的供给。

但是这里我们没有解释什么是政府债券？什么是债券收益Yield? 债券是当政府、企业、银行等机构需要向社会筹集资金的时，向投资者发行，承诺按照一定利息并约定条件返回本金的债权债务凭证。可以理解跟民间借条一样，约定借的钱，利息，返还时间收到法律保护。债券是一种债务的证明书，而股票是公司为筹集资金而发行给各个股东作为持股凭证并借以取得股息和红利的一种有价证券。简单说债券是保证返回的，相对来说风险更小，但是一般来说回报也肯能比股票的高或低，因为利息是固定的。简化解释来说：债券风险小，回报低；股票风险大，回报高。债券跟借条一样需要你对发行人有要求吧，这里面政府一般来说最可靠的，所以政府债券government bond就是因为政府需要支出而发布的债务证券，普遍认为政府债券的风险是最小的，相对于比如公司债券，公司股票等等。政府债券有分不同的类型(Bonds vs. Notes vs. Bills)，按照到期时间的长短(Materity)的不同，分为比如短期3个月一两年，和长期10年或者30年，一般来说他们的回报率都是跟着到成熟时间成正比的。

回报率(yield)等于每年的利息收入除以纸面价格，假设你第一天去买了一个政府债券纸面价格标注(face value/par value)的是$1000，市场竞价之后也是$1000(market value)买到，每年支付利息收入(coupon)是$60，那么回报率就是60/1000 = 6%. 但是第二天这个纸面价值$1000也是会在交易投资中变化（市场价值），如果价格下跌，比如$950，那么你这个时候买新的就有60/950=6.3%，回报率上升。这个有什么关系了？如果我只能持有到到期结束，但是债券是可以转手买卖的(transferable/tradeable)。也也是说我第一天买的实际上相当于没有保值，通货膨胀了，债券的价值降低了，毕竟拿到了同样的利息收入，但是第二天购买的人少支付$50；相反如果第二天债券价格升高，也就是新买的人支付更多比如$1050，那么相对来说回报率就会降低(5.7%)，也就没有那么吸引人去购买，换句话说老的持有者的价值升值；

图片来源： https://www.finpipe.com/bond-teeter-totter/

回到为什么长期债券的收益率大于短期的？看看财政部的每日收益率曲线Daily Treasury Yield Curve Rates这种表里我们到2021年1月21号的，一个月债券收益率是0.07%，两年的是0.13%，十年是1.10%，三十年是1.85%。也可以理解的，如果我短期借给别人钱，那么相对来说借的成本相对长期借贷来的不高的；一个借款7天，跟一个借款30年，肯定长期借款的风险更大，你不知道经济情况会怎么样未知因素太多有没有黑天鹅，另外也有机会成本（这中间可能有其他更高收益率的投资-比如买彩票，但是钱被锁住了），所以长期借贷的利息（回报率）相对来说要高点，而且短期一般来说，风险更小些，更可控些利息（回报率）低一点；另一个我们结合上面的描述，假设一个持有2年债券的人，跟持有一个10年债券的人，现在过来一年半，然后政府某种操作提高了债券收益率或利息，也就是这两人持有的债券的价值都贬值了，新买的人用更少的钱获得同样的回报，这个时候2年的人可以只用等半年就可以重新买更值钱的债券，但是这个持有10年的人要么等8.5年买新的或者打折卖出自己手中还没到期的债券，这个时候持有2年的人就比10年的人更灵活；

有一个具体的案列更契合能理解这个收益率，就是1951年的《财政部和联邦储备系统协议Treasury-Federal Reserve Accord》,该协议终结了美联储支持美国国债价格的义务，确保了联储制定和执行货币政策的独立性，把美联储从上面提到的1934年罗斯福设置的“财政部的提线木偶”的角色中拯救了出来，从此走上了独立自主的道路-后续影响很大（虽然没有到1980年沃尔克那样的独立程度）。从官方的历史资料来看，主要的冲突就是战后美联储一直也都是听命政府二战时需要降低长期国债的利率（回报率）来帮助政府低成本的筹钱，但是我们知道利率低，钱太多，那么相对应的通胀率也开始上升。1946-1948年的通胀率一直保持在高位，从17.6%下降到9.5%，但是还是接近双位数的水平，美联储认为他的目标已经从之前战时委身求全（因为战争失败，政府垮台，美联储也得完）的帮助政府筹钱，现在到了应该是战后限制通胀的本职工作了。但是总统和财政部部长为了保护那些战争期间买了政府的战争债券的爱国民众的利益，强力支持美联储保持低回报率。（这里就是上面讲的，如果利率回报率上升，那么那些之前战争期间买了10年30年的战争国债的，就贬值了，利益受损，那这些会想：凭什么我一腔热血省吃俭用那么困难的都要支持国家。

有兴趣可以看看那些战争债券War Bonds explained How children helped pay for both world wars的宣传和广告Promotion of war bonds in the United States during World War II. HD Stock Footage，到头来战争过去了，你们还要蚕食我们的利益。所以美联储可能知道自己很理性，但是有点无情，所以一直还是保持低利率；直到1951年朝鲜战争爆发，美国国内本土的通胀率高达21%，这个就让美联储坐不住了(untenable)，最终跟政府来来回回争论最终达成了协议。

At the time, it was not known how profound an effect that statement would have. But the accord marked the start of the development of a strong free market in government securities, which continues today. In addition, the debate over the consequences of interest rate pegging marked a shift in thinking at the Fed. Monetary policymakers began focusing actively on bank reserves and the control of money creation in order to stabilize the purchasing power of the dollar. But most important, by establishing the central bank’s independence from fiscal concerns, the accord set the stage for the development of modern monetary policy.

来自wsj: Why Investors Are Obsessed With the Inverted Yield Curve

为了方便理解，我们最好记住供给理论和风险回报比，还有衡量事物的尺度是否主观还是客观。

政府债券最安全，但是利率低，一般是如果没有其他更好投资机会（比如回报也更高的公司债券，股票，房地产等等），可能是因为经济危机或者贸易战或者比如911等事件，一般是经济萧条不景气的时候（风险>回报），他们其次才会更多考虑政府债券，此时购买的人需求多，供不应求，那么债券的价格就会提高，也就是回报率下降；相反如果经济情况很好，公司房地产铮铮向荣，那么你肯定会投入公司的债券股票等等，跟着势头走来换取可观的回报，这个时候低回报和没那么相对有安全（试想下多少个股票奔溃前夕的疯狂-人民日报说的4000点才是牛市的起点等等，情绪和贪婪让人的安全概念发生了扭曲)的政府债券没得人就少了，那么政府债券需求变少，无人问津，则价格势必要下降，对应的了回报率上升；

我们看到的正常情况下的回报率跟时间的曲线大概是上面这种慢慢上行的类型上坡的形状。左边底下是短期利率，美联储通过公开市场操作买卖短期政府债券，来影响短期利率，但是不会很快传导到长期利率。长期利率更多是投资者的感觉感受，正如上面说的，他觉得现在经济形势不错，那么不会买低回报的政府债券，而是会转而投入股票房地产等等，需求少->价格低->回报率高；相反，经济形势差下行，股灾或者房地产奔溃，公司大规模倒闭，则政府债券更安全，钱就少赚就少赚点，留得青山在不愁没柴烧，需求多->价格高->回报率低；

正常情况长期的比如10年的回报率是大于短期比如2年的回报率的，但是如果经济情况不景气，投资者的信心不足，可能会购买更多的长期债，造成长期利率下滑；如果经济情况得不到改善，则可能有更多人慢慢涌入这个长期债券，导致价格进一步升高，回报率下降，后买的人更加觉得要赶紧购买来保值；同时一般经济危机时美联储都会下调短期利率，这样一来也会使得长期利率慢慢更低；这个时候一般来说是两者双双下跌，但是某种情况下长期跌的比短期还多，也就是出现了利率倒挂Inverted Yield Curve，变成右边比左边还低的情况。普遍认为这是不好的情况，标明公司的融资的难度越来越高，这些金融机构的利润也会受损，更重要的是这个倒挂跟随后的历史上的很多危机都能对应上。下图是2019年2月份的来自WSJ：

当然经济学是一个模糊的学科，肯定不能靠单一指标预测的，没有银弹。

政府债券的价格Price和回报率Yield受什么决定了？它是如何售卖的了？

拍卖auction。

具体来说参考财政部的买卖指导：Treasury Bonds: How To Buy。你可以个人直接跟财政部购买，注册账号，然后提交bid出价投标。但是主要还是经过公开市场操作，包括银行经济商行等等，大家一起竞标购买或者出售，最后达成一个大家都认可的价格。然后才会在二级市场频繁交易和从银行或者经纪人那里购买。

比如美联储的公开市场操作，就是有公开市场操作委员会先开会决定一个目标基准利率值federal funds rate，然后去在政府债券市场和其他主要是银行竞标购买或者出售，达到预期的值。买卖的主要是短期的债券比如三个月等等，通过影响供给关系，提高或者收回银行的储备金，从而影响短期利率，调节货币供给。比如如果美联储觉得经济不行，像08年经融危机，公司融资没有了地方能借到钱（1: 食品用光了，没吃的了；2: 食品被其他藏起来了，觉得你风险太大，不借给你），这里假设1，银行是有偿付能力的，但是短期收不回来的，那么美联储需要主要额外的钱也就是流动性，就需要提高货币的供给，降低借款的利息，这个时候就需去公开市场，大量购买国债，将钱注入到银行，各个银行现在钱多了，这个时候就容易借给公司度过难关了。

美联储通过这个主要工具来控制通胀，简单的逻辑可以这么表述：

基准利率值太高->货币供应减少->借贷成本上升->经济活动减弱->需求减弱->价格下跌->通缩
基准利率值太低->货币供应增加->借贷成本下降->经济活动增强->需求增强->价格上升->通胀

具体数字和变化可以查看FED官方的FOMC’s target federal funds rate or range, change basis points and level每年的变化，最近一次调整是在2020年3月16号调整为0.0-0.25%，非常接近0了, 2020年的通货膨胀率大概是0.62%。

为了救市，伯南克在07年九月份到2008十二月，把基准利率从5.25%一下子降到了接近于0。美联储不断在公开市场一级市场prime dealer（主要是几个大银行）购买短期国债将reserve钱给到这些银行。还记得之前讲的主要银行需要在美联储储备银行里保留一定比列的准备金reserve requirement基本是10%（这里面实际比这个复杂），超过10%的叫做多余储备excess reserves. 基础储备（低于10%的部分）是有利息(interest on reserves - IOR)，然后2006年制定法规规定了多余储备也会利息(interest on excess reserves - IOER)，美联储通过指定这个利息值可以调控货币，也是货币政策的常规工具。这几家垄断行一级银行拿到钱reserve notes，会去更大的二级市场里去购买给比如其他小银行，金融机构，对冲基金和个人等这些人手中的国债，从而将钱从美联储转移到更大的人群中，这样就实现了货币的供给，这些人可以用来投资更有前途回报更高的领域比如股票公司债券等等，方便社会融资和经济的发展。毕竟银行靠什么赚钱？放贷。

如图所示，美联储向一级银行购买国债的利息一般来说没有一级银行向二级市场收的利息多，所以一级银行一般来收乐意转手赚取利息差，同时还能配合美联储的工作。但是如果经济不太行的情况，比如美联储已经将基础利率降到了几乎为零，一级银行已非常低的成本拿到了很多钱，但是还有一个问题，我可能拿够了补充基础储备的数了（达到了准备金利率），那我为什么还要多付利息去拿更多钱的了？因为外面风险大回报低，还不如拿在手里，如果可以，为什么要付多余的借钱的利息，虽然很低了。所以美联储可以相对而言提升下储备利息，这样银行有动机拿更多的钱在手中，这样也许可以增加比如REPO或者社会临时需要用钱的渠道。

这就容易陷入流动性陷阱Liquidity Traps：这些储备金的利息可能高过短期国债的收益率。比如2017年5月19日这天，一个月的债券收益率是0.71%，但是IOER多余储备的利息是1.00%，也就是说一级银行更愿意拿着储备金放在那里，也不愿意去二级市场买社会大众的债券，因为多了0.29%的利润。为什么了？因为一级市场只能是限制性的几个大玩家，而债券是更为广泛的被人持有而且可以当做抵押物向别人借钱（这个就是上面的REPO买断式回购，短期资金周转，需要借钱可以拿国债抵押，承诺第二天用稍微高一点点的价格赎回来）-这些特点是储备金没有的。虽然举的例子是2017年的已经是QE之后的几年了，但是这个现象也还是QE之前某些情况也是会存在。

要参考REPO买断式回购具体怎么运转，推荐看看WSJ的[2019 - Repo: How Roughly $1 Trillion Moves Overnight

WSJ](https://www.youtube.com/watch?v=gzCkXNrjFQM&ab_channel=WallStreetJournal)。基于这种情况，稍后提到的2020年危机了美联储很创新的立马把IOER多余储备的利息从2020年3月份1.6%降到了0.1%，目前2021-01-25这个时候还是0.1%，而1月期国债收益率是hmm， 0.07%，还是有0.03%的利息差，不过很比2017年的0.29%还是小很多的。

回到正题，结合这张短长期利率图表和伯南克的如何应对危机的讲座视频和PPT(Chairman Bernanke’s College Lecture Series - Lecture04 - The Aftermath of the Crisis):

鉴于短期利率已经到底，常规武器已经用完，但是经济形势还未好转，担心通货紧缩的风险加大。美联储换了思路，借鉴了前期日本的非常规武器的实验的经验，决定开启目标式的量化宽松。

先看看形势：08年金融危机是次级贷引起的，Mortgage-Backed Security，抵押支持债券或者抵押贷款证券化。而房贷一般属于长期的借贷，还包括比如车贷等等都可以认为是长期贷款，一般时间是比较长的，这块是收到打击最严重的。危机之后房屋的价格暴跌，很多在建的楼盘停止，很多建筑工人失业。这个时候资产价格下跌，但是债务并不会缩水。

我们看到了平常情况下没问题，但是现在价值100w房子假设暴跌只价值$1(极端点方便解释)，这些不良买房人因为利息升高等等每个月已经越来越付不起利息（或者现在房价下跌的如此厉害，我本来流浪汉，要赔钱，甚至房子不要了，跑路），自然这些购买MBS的投资者就会有损失，最后去找银行，银行说那我回收房子了吧，结果房子现在去卖只有1块；银行跟投资者说跟我没关系，你们找评级的保险的都行，这些人本来能赚钱就是基于房价上涨的预期（击鼓传花），现在这些多来找理赔的，问题是我也投入高风险投资了，也没钱了，不知道找谁。所以这些投资者的损失谁来赔付了？OK，假设这个不良买房人比较有良心，收入有只是比较少，还可以慢慢付，但是肯定心里不愿意了，因为房价是跌了，但是债务没减少啊，利息还得交，如果是那种可调利息，后面越来越高，那他的损失谁来保？美联储说这个MBS，保险，银行环节我来兜底吧。

量化宽松的时间线：

关于量化宽松是什么？为什么有宽松和效果怎么样？我们可以看看先行者日本央行BOJ的在2002年《The Challenges Facing Japan’s Economy and the Monetary Policy Response》会议纪要有一章节特别讲到QE的由来，如何工作的和效果A. Purpose behind Quantitative Easing Measures and Their Effectiveness.

美国的量化宽松（用伯南克说的Large-Scale Asset Purchases大规模资产购买）第一步：直接购买抵押支持债券（MBS）- 不是常规的短期国债，也不是国债了，也不是公开市场了。

2008年11月，美联储购买了总计8000亿美元( $800 billion)的MBS，这是第一轮量化宽松。直接去美国政府赞助企业(GSE，Government Sponsored Enterprise)的房利美和房地美两家非银行住房抵押贷款公司购买MBS（其实是更广泛的按揭相关的证券，这里简单些MBS），直接使得长期的房贷利率下降到6%(Mortgages head down)以下，同时也带动了30年国债利率下降。

逻辑是：按揭利率下降有利人吸引人们买房，尽管此时需求肯定下降很多，同时因为美联储在市场上买走了这些相对风险高的不良证券，使得投资者也老老实实开始接受了新回报的其他投资。这样比较低的长期利率有利用降低借钱的成本，同时因为上面的原因，因为国债和MBS都被美联储买走，市场供给下降，投资者被迫去买其他资产比如公司债券股票等等，公司企业和社会的融资筹集资金的成本和难度就下降了，有利于刺激经济。

2009年3月，购买3000亿美元的长期国债，美联储的资产负债表超过4trillion.
2010年11月，鉴于高失业率和经济恢复缓慢，第二次QE2 - 有争议行的购买6000亿政府债券。《Federal Reserve to buy $600 billion in bonds》
2012年9月，宣布QE3计划，每月购买400亿美元的MBS。《Fed to launch QE3 by buying mortgage securities》
2012年12月，鉴于失业形势，宣布QE4计划，每月购买450亿美元国债。《Fed sets jobless, inflation targets; to buy bonds》。

QE3和QE4跟QE1、QE2区别在于每月购买，没有设置一个资金上限，伯南克考虑最多的是通胀，失业和经济形势，直到他觉得可以。

鉴于伯南克的历史，他最怕就是通胀和通缩，但是不断的购买增加货币供给理论上是会引起通胀的，所以这个也是他会经常被别人质疑高通胀的风险时，但显然失业率是更重要的课题。2013年4月Bernanke: U.S. economy ‘not out of woods yet’:

“So far, we think we are getting some effects,” he said, adding, “It’s kind of early.”

“We have succeeded in bringing down long-term rates down pretty significantly,” noting 30-year fixed mortgage rates are at about 3.4%.

Bernanke downplayed concerns that the Fed’s easy-money policies eventually will set off inflation that could derail the recovery. “I don’t believe significant inflation is going to be a result of any of this,” he said.

“If some really good news comes in, if you’re using a date, people wouldn’t know how to adjust that,” he said. “But if you’re using guideposts in terms of inflation and unemployment, then investors in the market can say, ‘Well, the date seems to be a little closer now than we thought.”

现对于给一个具体的日期和资金上限数目不同，他是以通胀(核心CPI回升到2.5%以上)和实业率(降到6.5%以下)为导向的（参考FED官方的”Transcript of Chairman Bernanke’s Press Conference December 12, 2012），这样他认为更利于投资者更好判断经济和稳定的预期。他跟他崇拜的导师paul volker沃尔克一样觉得一定要提供一个稳定的预期，不稳定的货币政策会影响人们的商业运作，这个可能比美联储在实际中各种操作效果好的多 - 比如沃尔克之前1980之前美联储的左右摇摆stop-go货币策略。

2008年QE的效果

2014年10月，QE4结束QE4 Explanation with Pros and Cons:How QE4 Changed Fed History。

2012年公开讲课时伯南克的总结：

长期利率下降 - 30年国债下降到了4%；公司信用得到了恢复，借钱成本低，股票价格上涨
帮助经济恢复，虽然房地产市场比预期的弱。
美联储的信用和对于稳定价格的长期的承诺（通胀率和失业率目标）帮助稳定了通胀的预期，是的通胀率比较低但是不至于通缩。
美联储的资产购买不是政府开支，因为这些资产最终会买回到市场（ps:理论上经济形势转好了就慢慢卖出 - 危机稳定之后伯南克中间一度卖出提高利息但是因为欧债危机撤回了）
更多公众沟通更透明 - 定期开会(Communication and Monetary Policy)公开美联储的想法和策略，帮助大众和商业理解未来的政策方向
但是GDP增长，房地产市场，房价和失业率都还是恢复到预期的效果，人们的信心还是不足。
危机中暴露很多监管的空白和问题，加强了金融系统的监管

危机中暴露的问题？

投资银行等等其他机构配合银行，保险，评级机构等等设计复杂的金融衍生品，同时过渡的使用杠杆，使其获利。杠杆越大，收益也越大；同时也面临高风险，一旦出现问题，则杠杆会放大风险，使得公司陷入财务困境甚至破产等等，同时因为其金融机构之间复杂的交叉联系，非常容易放大到整个经融市场甚至全球，而这些机构最后被政府救助，大而不倒。什么是杠杆了？

上图看，假设我有2W块，如果我不用杠杆放大，那么投资的股票或者其他一般情况下比如上涨5%或者下跌1%，我只能得到其对应的回报或者损失，也就是+1000或者-200。如果这个人使用了杠杆放大了倍数，比如5倍，那么股票还是上涨5%或者下跌1%，但是其回报和损失都会放大5倍，也就是+5000和-1000. 借钱给这个人放杠杆的人一方面收取一定利息回扣，同时如果股票下跌，他会确包在亏完本金的时候叫停交易，比如如果亏到了2W，那么放杠杆的人会强行平仓，这样他借出去的钱有保证。

这样一来看起来双赢：杠杆人可以（比如澳门线上xxx开业了）以小博大，紧张刺激，能挣得更多的钱；借钱给杠杆的人可以拿到回扣利息，哪怕一旦下跌，他还可以强制平仓保证自己借出去的钱-本金；

这样一来，只要上涨的概率大于下跌的，那么加杠杆的人赚钱的机会相对来说会大一些；特别是如果股市房价一直上涨，就比如从2000年到2006年房价涨的越来越快，这个时候我越加杠杆我赢得越多，而且这种一直上涨的幻觉经过长时间的熏陶，你会认为是永恒的，也就是下跌是微乎其微，甚至不可能，那么不加杠杆才是傻了；看看索罗斯做的，不是啥把鸡蛋放在篮子了，这句话上下文不全，当你确定机会的时候就应该下最大仓位。就像索罗斯徒弟徒弟Stanley Druckenmiller德鲁肯米勒在狙击英镑中被索罗斯反问你把这也叫做仓位?：

从投资组合的角度来看，德拉肯米勒这样说是为了达到真正杰出的业绩，你在行情顺利得心应手地交易时，要充分利用当前的优势。那时是真正需要你紧握桂冠不放的时刻。杰出的业绩纪录可以避免出现亏损年，要尽力创造一些每年收益率达到二位数以上，乃至三位数的年头。对于个人交易的情况来说，管理好资金意味着当你处于不常遇见的好机会，有深具盈利信心时，就要敢于投入资金操作甚至不惜用杠杆作用借入资金。就像在1992年著名的 “英磅战役”用20亿美元压下了100亿多美元的筹码。正如斯坦利.德拉肯米勒所说的那样：“当你对一笔交易充满信心时，就要给对方致命一击。索罗斯对我不多的几次批评是因为我对市场判断正确时，没有最大限度地抓住机会放大胜果。”

机会确定时候上杠杆获利丰厚；但是如果这个”机会“由主观的”确定“（年年都在涨，今年也一样）变成了客观的“不确定”，出现里脱节，那么这个效果就反过来了，一点点浮动都会被放大，甚至某些时候下跌幅度之大，你甚至来不及平仓，就已经亏完了所有的（自己的+借的钱），比如之前的中国银行的原油宝事件。这里我们也可以窥探到第三方银行机构是怎么卖给个人的，它宣传低风险（为什么了因为上面的讲的，如果油价下跌最差我们可以平仓，不至于亏本金），收益高，个人或者投资者也鉴于赚钱的欲望很难知道真实的风险(通常在签字协议的最隐蔽的角落）.

鉴于这些情况，政府觉得要加强对于金融市场的监管：Dodd-Frank Act多德-弗兰克法案于2010年通过，旨在对金融监管进行了全面改革，并对所有联邦金融监管机构和几乎所有美国金融服务业部门都产生了影响。监督管控更广泛的金融系统领域，提高银行机构的流动金要求，禁止银行相关联的机构（影子银行）从事交易自己的账号（Volker提出来的也称The Volcker Rule)，更多的压力测试要求更多的流动金应对不好的经济情况，避免大而不倒的问题-存款保险有了要求，要求衍生交易等透明性和标准。

QE实际的效果如何了？

上图里看到通货膨胀和m2的货币供应量没有明显上升，同时银行在储备银行的多余储备却激增。这说明什么了？我们看看之前BOJ在上面2002年提到关于A. Purpose behind Quantitative Easing Measures and Their Effectiveness中提到的：

The third finding is that abundant funds at banks do not necessarily flow into economic activity in the private sector, such as investment and consumption. The increase in the monetary base, or the provision of funds by the Bank, has not led to an increase in the money stock, which provide the basis for private-sector economic activity. Theoretically, an increase in the monetary base brings about an increase in the money stock((consisting of cash, demand deposits, and negotiable certificate of deposits)), because ample funds provided to financial markets stimulate the demand for money. In practice, however, signs that the transmission mechanism of monetary policy was functioning fully, such as an increase in financial institutions' willingness to lend and in inflation expectations, have been conspicuously absent. As I mentioned earlier, these effects are expected theoretically, but in practice they are not taking place.

It is, however, necessary to consider carefully the side effects of the quantitative easing monetary policy.

2017年stlouisfed的这篇文章Quantitative Easing: How Well Does This Tool Work?提到QE是充满争论，理论是含糊的同时经验主义的实际证据也是开放性的解释的（没有定论，一方面也是数据比较少）。文章提到从2007年到2017年，美联储的资产从$8.8千亿上升到了$4.47万亿 - 翻了五倍。按资产跟GDP比列来算，已经从2007年的0.6%增长到了2007年的23.5%。同时美联储几乎持有的都是长期债券或者MBS，没有多少短期的国债。

When QE is conducted in a system flush with reserves, the central bank is typically transforming long-maturity assets into short-maturity reserves. The key question, if we compare this to how conventional monetary policy works, is what advantage the central bank might be exploiting in conducting such a transformation. That is not clear.

就实际效果比较日本2013年之后的通膨率(抛开消费税政策）跟货币供给发现并没有提高通货膨胀率（日本目标是2%），GPD增长比较美国跟没有QE的加拿大比数据也落后，数据并不能支撑QE的理论上行的好效果。当然作者也说任何货币政策都是非常难去量化的.一种看QE的方式是QE就是把长期债券转变为短期储备存款reserve。这样的有两个问题：一个是为什么不是财政部多发性短期债少发长期债券这样也可以达到美联储的效果，这两个谁来主导？第二是私人部门能比央行更好将长期转成短期，而现在这群人拿着多余的存款准备(reserve)也不去房贷而是赚利息，何不使用一些更丰富的工具来踢踢他们屁股了？比如中国的中央银行票据。

尽管对QE有不同的看法，但是拉长点我们看2019年，美国GDP增长2.2%(一般， 08-19年平均大概2%)，通胀率1.81%（08-19年总体不超过2%，还差0.5%左右），失业率达到了1969年来的历史最低点3.6%(08-19年一直在稳步下降)，道琼斯工业指数26,379.55点（十年一直牛市上涨两三倍)，基准利率也上升恢复到了最高2.42%。从美联储官方统计的这张表看Total Assets of the Federal Reserve ，美联储资产从2007年的8.8千亿到2017年巅峰4.78亿到2019年已经回落到3.75亿，应该说经济有向好，美联储也开始卸掉自己的债务。

鉴于这些数字来看除了GDP增长慢了点，其他数据都很漂亮。美联储主席也已经来到了杰罗姆·鲍威尔（Jerome Powell）。

[2020年关于现代经济的一些思考(四)]

2020年关于现代经济的一些思考(二)

2021-02-02T14:55:32+00:00

1907 Panic恐慌

镀金时代的美国是一个野蛮生长充满腐败缺少监管的时代，美国内战之后百废待兴，林肯的政策主要是修建铁路，但是因为铁路是相当于今天我们中国半导体行业，领先的是英国，需要引入资金，技术和人手，这些就不能是分散纷繁林立的小公司能实现的，加上铁路运营成本，恶性竞争，没有统一的标准，造成了极大的浪费。所以摩根借助金融手段，引入欧洲的资本投资，购买更好的设备科技技术，同时整合铁路行业进行垄断，一方面确保投资人的收益可以源源不断引入更多的投资驱动开发，一方面有效的提高了效率和实际铁路行业的前进。所谓的摩根化：

“摩根化”（Morganization）是J.P摩根在19世纪使用的方法，J.P摩根接管一个行业并用垄断来稳固该行业，再用自己的影响力吸引欧洲金融家进入美国。然后摩根将行业打造为一个单一的、稳定的、高利润的统一体，以吸引欧洲的银行家。(资料来源于—Morganization)

Morgan reinvented how monopolies can be created by eliminating competition through buying up smaller companies, decreasing prices until the competitors went bankrupt trying to compete, buying up the bankrupt competitors to cover more ground in a market, and slashing the workforce behind the company while reducing wages. Collectively, these actions maximized the monopoly’s profit. Morgan eventually took control of three major industries: railroads, electricity, and steel—and his dedication to efficiency and modernization revolutionized American business.

摩根是金融大亨，他在铁路、钢铁、电气等行业中建立了垄断，整合资源降低无需竞争来提高行业的利润，而这些正是银行的收入来源.

I like a little competition but i should rather have co-operation

为什么要讲到摩根？1913年创立的美联储跟1907的大恐慌(1907 panic)有直接关系。20世纪初美国经济进入一轮快速上升期，电力，汽车，铁路等等工业领域快速增长，经济的繁荣让企业迫切需要资本。一方面还是欧洲的资本被用来投机，即使这样也还不够就出现了信托公司，大肆向社会吸纳资金（类似投行），投资于高风险高回报的行业和股市。缺乏的监管的金融创新立马肆意飞奔，很多银行的贷款被信托公司抵押去股市和债券上投机。好比我们从余额宝，银行甚至高利贷那里借钱，去股市投机，跟2007和2015年国内牛市差不多，肆意上杠杆。一旦形式变化，英国格银行因为太多资本流出（英国的利息低，收益率不高yield,资金会转而留到收益高的海外市场-美国）转而提高利息，导致借钱成本升高，在美国的资金失去了一个来源，很多钱回流到英国（类似1970-1980年代的拉美危机）；另外旧金山大地震导致钱从纽约转出来用于旧金山的重建；过的投机倒把加上形式直转下，导致了股市的多米诺效应，信托公司破产，股票证券大幅下跌，无力支付银行的欠款。只要听到一家银行被别挤兑，所以人们听到任何风吹草动，都会被感染了一样一家一家银行挤兑，导致大量银行无力支付而宣告破产。

《1907年大恐慌》（The Panic of 1907）一书这样评论当时的情景：

“全世界处在经济崩溃的边缘，风暴中心是纽约金融区。尽管当时美国企业盈利迭创记录，整体经济持续增长，但美国股市却一泻千里，股价大幅跳水，券商纷纷倒闭，利率一飞冲天，纽约市政府无法发行债券，纽约证交所差点关门大吉，银行挤兑风潮席卷全美。”

(ps:2021-01-05日Jeremy Grantham发布了股市泡沫的文章WAITING FOR THE LAST DANCE，有人反驳说目前几个乐观点：一是美国企业盈利应该有所恢复；二是拜登新的财经刺激方案；三是疫苗进一步发放。实际上这些观点都不一定可靠，比如这里的企业盈利。)

政府的财政部（Treasury）也是面对不对恶化的形式无能为力(镀金时代信奉的自由经济，导致了资源的集中和垄断，即使后面有罗斯福和格拉斯-斯蒂格尔银行法Sherman Antitrust Act的拆分垄断巨头托拉斯，这些垄断巨头的影响力还是很大的）。为什么政府的财政部都没办法？我们看看银行和华尔街的情况，特别是摩根(引用自百科实际可以找找wiki或者其他数据):

1912年，摩根财团控制着53家大公司，资产总额127亿美元。这些资产中，有金融机构13家，30.4亿美元；工矿业公司14家，24.6亿美元；铁路公司19家，57.6亿美元；公用事业公司7家，14.4亿美元。

摩根死后自己的财产不多，但是他的集团掌控的领域和公司特别多，即使后来1933年分拆之后还有现在的Morgan Stanley摩根斯坦利和Morgan Chase摩根大通(后者又益于90年代金融送管克林顿-兼并）. 再一个摩根在1893年时也处理过经济危机（1893年危机中他帮助拯救了美国财政部-Panic of 1893），他应该说经验丰富，同时他利用自己的影响和其他巨头一起联合组成了高级董事会交叉控制网（interlocking directorates），为自己取了更多利润，控制合并了非常多的大工业和铁路领域，和其他巨头一起形成了影响力非常巨大的Money trust(金融托拉斯)的超级企业巨头联盟，可以让他们控制了国家头部的大银行和其他金融机构。所以目前谁有最多的钱来给市场注入继续的流动性了？不是小老弟财政部，而是私人巨头们。

1907年应该是摩根最风光的一年。老头子当年已经是70岁，应该说见过了不少美国经济的繁荣和危机间隔几年十几年的轮回繁荣衰退周期boom&bust cycle（有兴趣可以看看，尽管美国工业腾飞，但是美国地大农业还是占有相当大比重，每年的农民的收货季节总能引起信贷市场的波动swing，而且美国这期间一直没有中央银行，加上其他因素投机，缺少监管等等）-，做了几件事情了:

分析已有的金融形势，看看那些能银行能保住(bailout), 哪些只能是破产（Bankcrupt)；
开源 - 钱从哪里来？摩根自己掏腰包出了很大一部分钱(我找不到具体多少钱)，和其他纽约的银行家商议包括美国财政部秘书长将政府的钱也存入银行来注入流动性（两天后财政部长科特柳Cortelyou存了25百万；洛克菲勒，美国最富有的人往银行存了10百万）；纽约证券交易所（New York Stock Exchange）通常需要银行放短期贷款来维持日常的股票交易，但是在人人自危的情况下，即使银行有钱也不敢随意借出去-风险太高（从个人角度可以理解，兵荒马乱的时候，就不要考虑能否挣钱，而是考虑保命拿住钱的问题了，参考日本）。因为没得到银行的钱来维持正常运转，纽约证券交易所主席不得不急匆匆的跑到马路对面摩根的办公室告诉摩根他可能不得不提前打烊关闭交易所了。摩根再一次召集了纽约银行的主席们筹钱，扭转了形式，应该说救了纽约证券交易所一把，尽管很悬但是避免了形式的进一步恶化（1913三月摩根过世时遗体从意大利运回经过纽约时，华尔街降半旗-通常只是给国家元首，纽约证券交易市场关闭了两小时）；
节流 - 摩根和其他银行家的钱不是很够了，财政部也是，这个无底洞的填补一定要恢复公众的信心，减少流动性的损失，连欧洲最著名的银行家罗斯柴尔德Lord Rothschild都给摩根表达他对摩根的“admiration and respect”膜拜和尊敬; 随后金融市场恢复了平静，实则还没渡过风声鹤唳草木皆兵，一个经纪行用田纳西矿业和制铁公司的股票质押借贷然后用于其投机导致亏损，银行都逼迫他卖出田纳西矿业和制铁公司的股票来还债。摩根认为这个时候逼迫卖出这么多股票（本来就交易量小）来清算，会导致股票价格大幅度大跌（好比07-08年次贷危机房价螺旋式下跌，没人买了，需求急降），造成本已经战战巍巍市场的恐慌。摩根一个方案是用自己之前帮助合并过的钢铁公司来收购，但是这个会触犯憎恨华尔街和大公司的罗斯福的反垄断法。最终罗斯福在衡量不接受合并可能股市奔溃的后，同意了合并，摩根以超低的价格收购了TC&I，危机终于彻底过去了。

来源: http://santarosahistory.com/wordpress/category/bank-panic-1907/

1907-10-23号在上图中财政部长Cortelyou出来安抚公众说”形势好转了很多“，右下角说道信托公司有能力满足他们存户的兑换要求（on-demand)，而且这个是因为群众听信谣言说公司处于困难的境地，民众的信心在休市的时候得到稳步的增长（实际还是只是一个开端，有点像当年钱宝2017年8月份有一次挤兑危机，结果钱宝出来说大家不要听信谣言，随时来取款信誓旦旦亮出自己底牌多丰厚，结果借新还旧漏洞补不上半年后倒了，所以这个有点回光返照的意思）

1907-10-26号摩根经常开会商讨研究从晚上一直到了早上4-5点，有时候太晚，有时候太早，坐车时太冷甚至乎的了感冒(cold)，但是媒体评论正是这一寒颤却稳定了市场。

来源：https://www.youtube.com/watch?v=6Rb2TsOm3Kw&ab_channel=CNBC

摩根几乎是一己之力把美国从经济危机中拉了出来，一时间成了备受赞誉的国民英雄，但是似乎没有持续多久。

股市崩盘恢复可以很快，危机可能可以很快平复，但是恢复的时间特是异常的漫长和痛苦。一个是国家没有中央银行，缺乏经济调控的工具，货币政策不灵活，导致经济不稳定和僵化；出了危机，只能依靠一个私人银行家(可以说是非正式的中央银行行长)来拯救这个令公众和政府有点担忧，头部这小撮人居然有这么大权力，没有被管控；摩根救市也不一定出于爱国，他可能也知道金融情况恶化下去对他而且也是不利的，可能部分也是出于自利之心，另外危机中无底洞一般的资金投入也让他们害怕，害怕再一次重新这个现象.(1907 - back when bankers bail out gov, not gov bailout bankers now)

来源： Meaning and History of the Term Robber Baron

政府和公众认为这小撮人-金融托拉斯-踩着人民的身上，通过金融手段合并和通过挤压、低价倾销等方式拖垮还活着的对手（参考洛克菲勒）从而获得垄断权力，控制了物资价格，同时不断延长底下员工的工作时间和无视恶劣的工作环境（童工等），从而积累了大量的财富，被人们称为”Robber Baron强盗式资本家“。这群商业金融资本家就和底层的农民等等形成了矛盾，农名和社会的情绪过去之后就觉得金融危机可能是这群Money Trust金融托拉斯自己玩脱了或者自己一手设计的，从他们和信托公司（吸收了大量存款）中手中将辛苦钱骗走（1907年大量大量银行倒闭-倒闭数量历史上第二大，导致普罗大众的存款消失），还有就是比如摩根超级低收购田纳西矿业和制铁公司这个事情，肥了摩根这群人的钱包。

政府和公众盯上了这小撮子人。

1912-1913年民主党员(内战赢了的林肯以来简单说共和党代表了汉密尔顿的工商业派-北方，民主党成了托马斯杰弗逊的农业派-南方没落之后1861-1912期间只有两任总统)阿尔塞纳•普约(Arsène Pujo)举行了普约听证会Pujo Committee质询Money Trust金融托拉斯-摩根和纽约这些有权利的显赫的银行家们有没有滥用公共信任来巩固加强联合控制多个工业领域。他发现摩根公司的高层也是其他112个公司价值225亿的高管（当时NYSE总市值也就260亿). 摩根和这些人 - 私人企业家们 - 可以说实际上控制了美国这辆车的车头。

其中有一段是摩根跟一位以严格犀利风格出名的律师Untermyer的character个人品质品格对话：

Untermyer: “Is not commercial credit based primarily upon money or property?” (Untermyer: 难道商业信用（指商品交易经营活动之间相互提供的信贷。如延期付款、预付货款)不是主要基于金钱或者所有物么？ )

Morgan: “No sir. The first thing is character.”（摩根：不。第一位的是个人的品质）

Untermyer: “Before money or property?” （Untermyer：甚至在金钱或所有物之前？

Morgan: “Before money or property or anything else. Money cannot buy it…because a man I do not trust could not get money from me on all the bonds in Christendom.” （摩根：在金钱或所有物或其他任何东西之前。金钱不能买到它…因为一个我不信任信得过的是不能在一切基督教国家从我这里拿到钱。

上图是朋友圈老家的一些朋友有时候过年过节都会发的，他们一般在老家都做点小生意。信用这个东西很特别，但是这里我们看到一般人与人之间信用说到底还是看这个人怎么样；如果他有上进心，你熟悉了解，尽管目前可能有点困难，可能你还是会借给他，即使超过他此时此刻能还的数目；如果你知道这个人不咋样，还信任他，鼓励借给他，那么你要么是高利贷做陷阱让他陷入更大的债务（要么砍头息-多看看今日说法，参考这一期《20210110 被盯上的拆迁户》；以老家的小圈子，记账也得都得是看人如何的。一般来说，个人信用很重要，国家的信用就更重要了。

不管怎么样，所以很快摩根和政府公众都至少有就一个问题达成了一致：需要建立一个类似欧洲的中央银行，虽然民众一直反对中央银行，就像早死的美国第一和第二银行.

1913 FED

危机中显示出两个问题：不够灵活的货币和没有足够的流动性支持。之后国会成为了一个金融专家为主的国家货币委员会，参议院的共和党人Nelson Aldrich是主席。Nelson Aldrich在参议院和共和党内部影响力很大，泰迪罗斯福称为共和党的KingPin，很多决策都是他和几个人（“Sentor Four”）完成的，应该说是国家的经理人（”general manager of the Nation)。他在参议院1890年跟商人合作搞有轨电车的生意成为了百万富翁，同时坚定支持金本位（利于贸易），他的女婿是洛克菲勒家族的，他一直被指责跟华尔街的金融银行家有密切往来。他游历考察了欧洲的各个央行制度，跟后面其他几个非常有影响力的银行家和工业联盟秘密的在杰克尔小岛（Jekyll Island）拟定了后来的《 Federal Reserve Act in 1913》法案的雏形。这个岛在18世纪后叶它是专供富豪们休闲狩猎的场所，像Rockefeller, Morgan, Frank A. Vanderlip(现在花旗银行) , Pulitzer, and Baker等都是狩猎俱乐部Jekyll Island Club的成员。一个是当时最有权力的政治家（有金融背景），一个是拥有当时1/4世界财富的金融银行企业家们，也解释了当时为么他们如此这么偷偷摸摸生怕被发现（因为一旦曝光，就没有任何机会可以在国会通过，因为民众像08年之后痛恨那群银行家华尔街一样的痛恨这群人）。

美联储官方关于Federal Reserve Act的正式标题：

”An Act To provide for the establishment of Federal reserve banks, to furnish an elastic currency, to afford means of rediscounting commercial paper, to establish a more effective supervision of banking in the United States, and for other purposes.“。

基本上就包括上面描述的紧急情况下向银行贷款，印钱和作为美国政府的财政代理人，监督银行体系。包括最上面的美联储委员会董事会（ Federal Reserve Board of Governors），和十二个地区来代表多样性的地区银行（ Federal Reserve Banks ）组成负责具体执行美联储政策。

那么当时的美国总统Woodrow Wilson(伍德罗·威尔逊)民主党人（民主党人几乎控制了两院）跟共和党人也就是具体起草人Nelson_Aldrich的矛盾地方在于什么？简单说公共vs私人。Nelson Aldrich最初起草的希望这个央行系统全部有私人（银行家和工商阶级）构成，不希望政府等公共的介入（independent)，独立的机构。但是民主党人更像代表了农民草根群体害怕央行变成这群华尔街银行家的money trust托斯的牟利工具，所以才有之前讲到的Pujo Commit的听证会。民主党人想要修改草稿来达到救济金融恐慌，减少失业率和商业萧条，最后保证大众的利益不被纽约这群金融托拉斯所绑架而被侵害（ps听起来很熟悉)。两方面都想争取的利益，经济政治因素利益的考虑权衡，远比我们想的要复杂。好比法案到了快要表决的时候，几乎打成了平手，一个来自密苏里州的议员的选票就很关键了，谁能争取下来谁就赢了。为了让方案通过，制定人私下承诺在这个议员所在的这个州联邦第八区（下图）开特列有两个联邦储备银行，即使这个州影响力远小于东部沿海，甚至弱于崛起的西部，到现在都没变 - 所以我们看到这个之间远不是一方绝对压倒性控制另一方，如果有那么只能说在1955年前都是政府压倒式控制美联储，美联储只是一个傀儡工具人.

来自：https://en.wikipedia.org/wiki/Federal_Reserve_Bank

结果就是总统可以认命美联储的董事会；而下面十二个区域的联邦储备银行(Fed Reserve Banks)负责一起实施货币政策，放贷业务和监督区域内的银行，清算，作为最终贷款人 lender of last resort。

联邦储备银行实施董事会的决策，他有私人公司的属性也有公共的比如政府的代表属性。每个区域下的商业银行需要持有联邦储备银行银一部分股份才能成为成员银行(Member Bank)，同时每个联邦储备银行的董事会成员分为三种类别，第一种Class A由底下成员银行选出，代表成员银行的利益；第二种ClassB也是有区域内的成员选出，但是应该suppose代表公众的利益；第三种Class C有由上面美联储董事会指任，意在代表公众的利益。这些成员银行虽然拥有对应的联邦储备银行的股份（相当于实力保证），但是这些股份是不能转让售卖的，联邦储备银行只有利润的分红才会给到成员银行 -一般是6%（公司越大，分红比列越少），跟一般股票不一样的他们不直接控制董事会（好比华尔街像是黑石BlackRock拥有很多三星的股份，但是没有董事会的投票权对公司具体运作不过问，只是投资获利而已）

参考U.S.-based Asset Manager BlackRock Boosts Stake in Samsung Electronics Above 5%:

Meanwhile, BlackRock has become the third largest shareholder of Samsung Electronics, following the National Pension Service with a 9.25 percent stake and Samsung Life Insurance Co. with an 8.24 percent stake. However, BlackRock is not planning to participate in the operation of Samsung Electronics. The company said, “We acquired the shares simply for investment. We will not carry out any activities to exercise our management rights or participate in the operation of the company.”

联邦储备银行他不完全像是商业银行，那么灵活，他的收入怎么来了？从官方网站描述《为什么FED跟政府是独立的了？What does it mean that the Federal Reserve is “independent within the government”?》提到：

The Federal Reserve does not receive funding through the congressional budgetary process. The Fed’s income comes primarily from the interest on government securities that it has acquired through open market operations. After paying its expenses, the Federal Reserve turns the rest of its earnings over to the U.S. Treasury.

收入来源主要是购买债券的利息。这些扣除运行开支，主要是返回给美国财政部Treasury，只有很少很少的部分才会最为利息分发给他的成员银行。比如下图中所示，收入和支出：

来源： https://www.frbsf.org/education/publications/doctor-econ/2006/may/federal-reserve-funding

我们看到网上说阴谋论是说FED美联储是被私人银行和企业家操控？一个原因是他自己网站Section 7. Division of Earnings上都说给持人6%的分红。分红我们比列我们大概讲了，记住只是利润，那我们看看实际中每年美联储赚多少钱，多少返回给财政部，多少用于分红了。

从这篇2012年的美联储官方的文章Reserve Bank income and expense data and transfers to the Treasury for 2011中，我们看到

The Federal Reserve Board on Tuesday announced preliminary unaudited results indicating that the Reserve Banks provided for payments of approximately $76.9 billion of their estimated 2011 net income to the U.S. Treasury. Under the Board’s policy, the residual earnings of each Federal Reserve Bank, after providing for the costs of operations, payment of dividends, and the amount necessary to equate surplus with capital paid-in, are distributed to the U.S. Treasury.

The Federal Reserve Banks’ 2011 estimated net income of $78.9 billion was derived primarily from $83.6 billion in interest income on securities acquired through open market operations (U.S. Treasury securities, federal agency and government-sponsored enterprise (GSE) mortgage-backed securities, and GSE debt securities). Additional earnings were derived primarily from realized gains on the sale of U.S. Treasury securities of $2.3 billion, foreign currency gains of $152 million, and income from services of $479 million. The Reserve Banks had interest expense of $3.8 billion on depository institutions’ reserve balances and term deposits.

In 2011, statutory dividends totaled $1.6 billion and $375 million of net income was used to equate surplus to capital paid-in

也就是(1.63/78.9)= 0.020 也就是2%左右的是用于分红。Who owns and controls the Federal Reserve 1995年几乎是97%的收入归回财政部，3%用于分红和其他。

The Federal Reserve System certainly makes large profits. According to the Board’s 1995 Annual Report, the System had net income totaling $23.9 billion, which, if it were a single firm, would qualify it as one of the most profitable companies in the world. How were these profits distributed? By an agreement between the Board of Governors and the Treasury, nearly all of the Fed’s annual profits are paid to the federal government. Accordingly, a lion’s share of $23.4 billion, which represents 97.9 percent of the Federal Reserve’s net income, was transferred to the Treasury. The Federal Reserve Banks kept $283 million, and the remaining $231 million was paid to its stockholders as dividends.

看起来分红的好像不多，那么分红不是一定依据他在美联储银行放的储备金（股份）的多少来的么？

从这篇文章中，举2018年纽约地区的美联储银行（New York Fed）为例，成员银行中Citibank花旗银行放了87.9百万，占比42.8%；第二名JP Morgan Chase银行，60百万，占比29%，两家加起来70%将近，其他的一级银行还包括比如日本和德国的银行等等。虽然理论上一行都是一样只有一票，但是上面说过他们有三个分类Class A, B, C，同时还跟银行体量大小分为：Group大(比如花旗和JP morgan Chase)，Group中，Group小三个级别的银行。上面说到ClassB理应代表公众利益，实际上却是Group大的可以选一个特别的指定的（designated）的ClassA和ClassB的代表，所以背后的选举程序是不是做到的公开透明难说，有没有更倾向大公司大银行的利益，这些流程是否有监管确实难说。

这些成员银行中有一些是一级大银行，成为Primary Dealers一级交易商。以纽约储备银行为列，你可以查到他的一级交易商列表: LIST OF PRIMARY DEALERS OF NYFRB。

在任何时候，一级交易商通常少于30多家，最新名单由纽约联邦储备银行维护，它处理公开市场业务，随后直接与一级交易商合作。作为一级交易商运作的机构通常在金融市场的其他领域非常活跃，名单上往往包括美国一些规模最大、实力最强的银行和经纪交易商的名单.它必须定期向美联储提交有关其金融活动的信息，包括交易量，并证明其维持着美联储设定的货币储备。它还必须同意参与国债的拍卖，以及在美联储参与公开市场操作时与美联储合作.Primary交易商是可以直接与美联储进行交易的金融机构一级交易商被用来承销新的政府债券，稳定经济，以及促进其他金融目标。美联储需要这些机构使其运作有效，而一级交易商每年的业务依赖美联储数十亿美元。一旦交易商购买证券，他们就可以转售给客户，或者在他们自己的行动中使用它们，在这一过程中，这些银行和经纪商往往能获得丰厚的利润，因为这些银行和券商的上市也能从中受益.

他们向是一张网将美国的国债出售到世界上的其他国家。一级代理商从储备银行那里借钱有一个利率，这个利率叫做Prime Rate(优惠利率). (这里没细分，你大概值这个是只有规模大信誉好的客户才能拿到的利率）。这个利率基于美联储的federal fund rate /美国联邦基金利率.

下面看看美联储的三个工具：

准备金比列 - 存款机构必须要保留一定比例的存款，不能全部放贷出去。假设我往银行存了100块，那么他可以放贷出去最多90块，还得保留10块，以防客户来兑换，这个10块就是准备金，也就是10%的准备金率。这个比列就是美联储决定的。
贴现率窗口Discount Window - 当银行被别人挤兑没钱了，这里没钱指的是不一定没有偿还能力（solvent）。比如上面里银行放贷了90块钱的存款，给了工厂去扩大产能。工厂可以拿比如已有的资产asset抵押，作为自己的信誉保证，获得钱来盖场招人，银行可以拿到利息，后面工厂之后得到了利润在还钱给银行，两者双赢，这个确实促进了社会的发展；如果银行信用卡放贷了给了某个人，某个人拿去全部身家投机上杠杆玩期货对赌比特币，然后比特币像是这两天突然下跌了10%跌破$30000美元(《突发闪崩，比特币暴跌12.5%，全网数字货币24小时8.6万人爆仓，51亿资金被“血洗”》)，直接爆仓，那么这个人肯定无法偿还了，银行也就是无清还能力insovlent; 不管那一种，银行现在都没钱了，抵押物要短期出手，肯定价格价值都大大折扣，但是每个人都不想成为最后一个去银行取钱的人（那时候没钱了），所以只能借钱，但是其他银行自身难保，就只能是求助于无中生有会生钱（当时还是跟金挂钩）的美联储了。这个利率就是贴现率 - 只在危机情况下作为最后的求助者。
公开市场操作open market operations (OMO) - 简单来说，FED的主要货币政策的工具。首先商业银行需要保持一定的准备金数目在美联储储备银行，而且每两周会审核。但是可能银行将钱放贷出去过多（临时），到了审核的时候，就需要往其他有足够钱的银行临时借钱保证审核过审，过审完再给返回去，这个借钱的利率是一个短期的利率，一般来说隔夜。美联储通过在公开市场出售和购买国债实现往经济中回收或投放货币。按照简单的供给理论来比喻，市场上钱多了，那么借钱成本下降，利率势降低了，反正利率升高。这样就可以通过直接影响短期利率，间接影响长期利率，从而进而影响方方面面，比如储存存钱的利率到长期的家庭房贷利率，信心卡的利率等等，更进一步影响信用的提供，投资的需求，就业和经济的产出（通货膨胀）。 2020年3月15日为了应对疫情，美联储下调美国联邦基金利率到了大概是0.00%-0.25%。

来自：https://www.youtube.com/watch?v=Vy6gSr2AtUE&ab_channel=MarginalRevolutionUniversity

回到这个prime rate, 美国38%的银行都是区域储备银行的成员，剩下包括民间的私人的小银行地域银行等等. 举例来说2020年中国人民银行公布的贷款基准利率是：一年以内(含一年)贷款利率4.35%，暴雷的蛋壳公寓的微众银行给个人贷款的利率是8.8%而且是消费贷（实际已经很良心了-蛋壳补贴），对于民企贷款利率10个点以上都很正常（还想贷贷不到）。对比一下京东的白条和阿里的花呗，简单的转换下日利息和每期费率（这些词容易迷惑，让人以为便宜，更利于冲动消费），套用下公式和计算，大概白条是18.4%和16.4%，实际利率不低，借呗的甚至接近民间借贷（规定应不超过26.24%）的利率。

所以我们看到利率有短期到长期的传导时间上，也有垂直方面的传导。这个传导的依据就是信誉值了。谁的信誉值高（大公司大银行），抵押物或者资产多，风险越低，那么能拿到利率就低；相反如果你经济实力不太行（个体商户），尽管可能事物有前途，但是风险高，那么相对应的要么你拿到利率高或者你让出的未来利益多（风投）；总的来说这套机制对于大的银行家，金融机构和大企业显得更有利，对于普通的个人和小公司小银行来说就不是那么友好。趋利避害，这套架构，某方面来说也是人性的一种体现吧，但是谁能知道当发生人性的扭曲道德的沦丧时会发生什么了？谁来制衡这套机制不会过度偏颇走极端了，大概率顶部的越来越集中了？

1929-1933 大萧条

帝国的荣光：1914年之前英镑是世界货币，英镑和黄金挂钩（实际是金块bullion)。英国是头号帝国主义国家，其他国家货币锚定在英镑上面，英镑和黄金绑定，形成所谓的黄金交换标准。英格兰银行也是欧洲各国效仿的对象，除了美国。黄金和资本的流动是自由的，使用同样的世界货币(英镑），拥有差不多一样的法律和监管体系（源自英格兰银行），国际间的贸易和投资变得非常容易。1914年44%的全球 Net foreign investment（外资）来自英国，20%来法国，13%来德国。这些投资都流向了当时的新兴市场（更高的收益率，上面提到的摩根）-美国（就像现在的中国）和拉美国家，非洲等。1870年到1914年，世界贸易跟GDP的比列从10%涨到了21%，这些投资推动了整个世界的发展。同时因为金本位带来的货币的稳定性，各国政府的长期债券的回报率都是比较低的水平- 跟今天央行各种骚操作黑魔法相比-当时的是比较真实反映了现实因为金本位支撑而长期稳定的宏观经济的乐观期望。

1914年欧洲大陆发生了一战，跟林肯一样，为了筹集足够的钱来应付战争的开销，英国暂停了金本位，暂停了英镑和金块的兑换，随后其他国家也跟上。

1918年一战结束，英国国内货币超发，通胀率居高不下，失业率下不来，欧洲各国开始也推迟了返回金本位的想法。1920初英国政府更想通过回到金本位来降低通胀率，而不是先考虑降低失业率。为什么回到金本位可以降低通胀率了？我们看下图一个简单的比方：

中国一开始因为货币少（这里假设不是美元或者人民币，货币指的就是黄金，因为货币锚定），价格低，不断生成商品，进而出口到美国；此时美国需要将黄金给中国，于是中国积累黄金（净流入国），此时货币>商品产出，钱越来越多，于是通货膨胀，人们希望今天就买东西，否则明天会贬值，此时造成工厂的工资和成本上身，进而再次带动价格上升，出口的吸引力下降；作为美国（净流出国）货币越来越少，人们手头上的钱变少了，购买的欲望下降，造成紧缩，工厂停止生产，进一步带动价格下降，到某个地步，生产商品的成本也会降低（相对而言），从而提高了出口产品价格的吸引力，形式又反转过来了；

某种程度上，金本位带来了一种自动平衡国际家收支进而稳定通货膨胀或紧缩的效果，缺点也很明显当经济出现危机时，某个局部或者国家没有掌控自己内部的货币体系，失去了灵活度，很难通过贬值注入货币供给，只能是通货紧缩财政收紧砍掉公共福利等开支（由奢入俭难），失去了国家部分主权，比如后来的欧元区的希腊的债务危机，到现在都有重回之前货币德拉克马的呼声，在Brexit单词的来源正是希腊脱欧Grexit.

英国想回到大战前英国在国际贸易上的统治地位，而这个统治地位是稳定基于价值围绕其他货币的兑换比例英镑的内在价值 - 购买力。

现在英镑大幅度贬值（超发），购买力严重下降，所以黄金的内在价值无法体现，就会流出英国，进入购买力强（保持金本位）的美国。现在英国英国不是为了抑制通货膨胀或者回到帝国荣光立刻回到金本位，而是效仿美国内战，慢慢的缩紧和稳定住货币量，营造一个稳定的货币环境，依靠经济发展的产出慢慢的提升货币的购买力，而不是立刻的紧缩。美国花了15年才回到金本位，英国1925年回到了金本位（大概是7年时间），不要忽略英国这次不是内战，而是欧洲的大战，战后各国都想以邻为壑比对方先恢复过来，所以外部形势更加恶劣。

1920年4月份，通过削减政府开支和提高利率导致了价格的下跌，伴随着的是大量的失业。1921年紧缩政策-大量的财政削减帮助减少政府的赤子，同时减少了英镑的流通量也使得英镑的购买力相对美元而言开始回升，但是也导致大量的罢工。1925年丘吉尔正式宣布英国重回金本位。重回金本位之后因为英镑的升值导致出口竞争力大幅度下降，大量工作机会消失，出现了一段滞涨经济衰退时期。欧洲其他国家就比较狡猾：一个原因是其他欧洲国家陆陆续续回归金本位，所以他们相对英国有对应的出口优势（货币贬值）；另外他们没有将兑换比例回归到站前的水平，实际要更低一点，造成了实际上相对英国仍是贬值，出口的价格更低，吸引力更多；比如法国积累大量的黄金，由于英国的影响力下降和目前各国互相冲突以各自利益为主，实际上回到了重商主义，直接也导致金本位的底层基础已经出现了动荡。1926年失业率激增，因为紧缩，更是基于对于将来的继续紧缩，英国爆发了1926年的大罢工1926 United Kingdom general strike，英国最重要的工业领域-煤矿工人组织工会要求政府组织工资降低和更恶劣的工作条件，更长的工作时间。英国这个情况令其他跟他竞争恢复的国家法国，德国，比利时等等很忧虑，因为他们手上持有很多世界货币英镑了，此时邻居英国看起来要跨了，感觉把英镑兑换成黄金靠谱点，于是争相挤兑英镑，造成了对英镑的信任危机。

英国差点没破产倒闭，没倒闭的原因是大西洋彼岸有一个被自己国会嘲笑为Eurocentric欧洲中心论的美联储主席本杰明·斯特朗（Benjamin Strong）。斯特朗也是当年Jekyll Island Hunt Club 的一员，参与了私密定制美联储草稿的会议。他极力想促成欧洲重返金本位，特别是对英国特别的上心，看起来更像以英国利益为先。1924年英国返回金本位之前，他就想通过降低美国的利率，通货膨胀，使得美元可以贬值，因为英国背负了大量的战争欠款（以美元为单位计算的，不是英镑），这样英国就可以减轻债务，以希望可以保持英国国内的紧缩的高利率，方便回到金本位。这是1924年斯特朗自己的话(Murray N. Rothbard, America’s Great Depression (5th edition), Mises Institute, Auburn, AL 2000 p. 147)：

…the burden of this readjustment must fall more largely upon us than upon them (Great Britain). It will be difficult politically and socially for the British Government and the Bank of England to face a price liquidation in England… in face of the fact that their trade is poor and they have over a million unemployed people receiving government aid.”

他的好搭档当时的英格兰银行行长Montagu Norman蒙塔古˙诺曼回复斯特朗是这样的(Murray N. Rothbard, America’s Great Depression, p. 147-148)：

You must continue with easy money and foreign loans and we must hold on tight until we know… what the policy of this country is to be.”

简单说请继续保持低利率的措施。但是这些还不够，1927年为了应对挤兑，斯特朗在长岛集合了世界上的主要央行行长们开了个会，主要是为了帮英国渡过挤兑危机 -主要办法是美国进一步放低利率，信贷宽松。这样的结果真是牺牲自己续命英国，这样的央行长真的少见。

1920年代是美国称为Roaring Twenties咆哮的二十年代发展飞快的时代，福特汽车，家用电器，洗衣机，吐司机，电影，电视机等等民众的生活水平大幅度提高，无法否认信贷宽松确实加速了这些进步。但是过度长时间的低利率，美国经济和公司利润的爆发，都让民众的消费观念扭曲和投机飞涨，憧憬股市将会一直再创新高，导致股票市场在20年代末期暴涨。

经济学家欧文·费雪Irving Fisher甚至立下名言，宣称“股价已经立足于像永恒的高地上”（Stock prices have reached what looks like a permanently high plateau）

这个开始让斯特朗有点忧心，但是他感觉别无他法-低利率是为了帮助欧洲国家（特别是英国）能重回金本位，回到当年稳定的金融货币体系。1929年斯特朗过世后一年美国发生了股市大奔溃，直接开始了多年的经济萧条。胡佛总统的顾问 Adolph Miller当时的FED成员之一责怪斯特朗的信贷宽松是：”father and mother to the subsequent 1929 collapse”.

来源:https://schiffgold.com/guest-commentaries/the-story-of-benjamin-strong-how-fatal-conceit-wrecks-economies/

有意思的：同样的美联储的主席格林斯潘在2000年dotnet泡泡之后开启了低利率，间接导致了2008的经融危机的发生。

1929年10月的股市大奔溃引起了多米诺效应，因为金本位的制度和美国当时的经济地位，实际上如同在水塘中投下了原子弹。欧洲银行陆续出现了挤兑，恐慌情绪漫步了整个欧洲，人们想兑换回来黄金和更强的外汇，是的各国央行的黄金储备不足。很快各国央行暂停了黄金兑换和外汇流通，同时各国又一次也跟群众一样迫切去英国（黄金中央银行）挤兑黄金。由于黄金挤兑和恶性套利Speculative_attack（可以联想到97年的东南亚金融危机），迫使英国1931年退出了金本位，其他欧洲国家陆陆续续退出了。金本位让各国在正常时期有了稳定的货币体系，促进了贸易的流通，但是到了危机时刻，这种共识就不复存在了，每个国家都自己为先，而金本位没办法让央行有灵活的货币政策，只能在情况最糟糕的情况下采取紧缩的方式（本应该印钱下调利率，贬值，扩大支出），进一步限制了经济复苏的可能和延长了时间。同时1930年的斯姆特-霍利关税法案（The Smoot-Hawley Tariff Act）更是让阻断了各国的贸易往来，恶性竞争和报复性关税，使得各国进入了一个恶性循环，全球贸易量大幅度减少。果然率先脱离金本位之后英国也是经济最快恢复起来的。

1932年《Glass–Steagall legislation格拉斯-斯蒂格尔法案》规定投资银行业务和商业银行业务严格地划分开，前者从事提存及贷款业务，后者则从事包销、发行及分配股票、债券及其他证券产品的业务(如银行可以自己给自己贷款来做空股票，可以将坏的贷款证券化再次销售给投资者，从而放大了金融风险。摩根银行被拆分为了摩根商业银行也就是后来的跟大通银行（Chase Manhattan）合并为摩根 Chase和投行Morgan Stanley摩根斯坦利。(此法案和合并的事情还在后面提到，现在美国最大的投银行就是JP Morgan Chase)

1933年大萧条之后的第四年，对于金融机构的不信任，鉴于群众挤兑的情况在美国也越来越严重，国会通过了《1933年紧急银行法案》（Emergency Banking Act of 1933），总统宣布了四天的银行假期（暂停兑换）清算银行资产，美联储承诺提供无限量的货币来支持重新开业的银行，银行储蓄保险FDIC(存款即使没了国家政府来赔），恢复群众对金融体系的信任。罗斯福利用电台收音机(根据这篇文章Radio: The Internet of the 1930s-1930年，40%的美国人有收音机，相当于今天的电视机和手机）来向全国人民说明国家的状态和政府的措施，帮助人民恢复信心，也被称为了Fireside Chat炉边谈话, 这个也是首创平民沟通方式。

“I can assure you that it is safer to keep your money in a reopened bank than under the mattress.” – President Franklin Roosevelt in his first Fireside Chat, March 12, 1933

来源：https://www.federalreservehistory.org/essays/emergency-banking-act-of-1933

同时法令也赋予总统在危机时刻脱离美联储能独立运作货币政策的权利 - 也就是现在总统可以直接做美联储的事情 - 管制外汇市场和黄金白银。

Expanded presidential authority during a banking crisis, including retroactive approval of the banking holiday and regulation of all banking functions, including “any transactions in foreign exchange, transfers of credit between or payments by banking institutions as defined by the President, and export, hoarding, melting, or earmarking of gold or silver coin.”

进一步的，1934黄金储备法案Gold Reserve Act国有化黄金,禁止私人持有黄金，美联储交出持有的黄金和金券给到财政部，禁止用美元兑换黄金，禁止黄金出口，确立美元和黄金的比列。随后罗斯福将美元和黄金的比列下调，从1ounce黄金对应20.67降到了1黄金对应35，这个比列一直持续到1971年。美元的贬值，禁止私人持有黄金，跟美国内战之后的废除银币是一样的，迫使你交出黄金，拿回美元，扩大货币的供给，有助于经济的恢复。

这些危机时的经济和社会改革的政策(新政New Deal)，可以看出政府的权利其实是扩大了的。我们看罗斯福在写给当时美联储主席时候的一封信中说委婉的说到他的政府的政策并不会干扰美联储的使命。但是后面前美联储主席提到罗斯福的信更像是一份悼词（来源官方历史的《Gold Reserve Act of 1934》：

“The plain and unvarnished fact is that the Federal Reserve System of today is not the one established 20 years ago, any more than it is the system which existed a year back. The present organization has been shorn of its power to formulate an independent credit policy and it can no longer regulate the flow of funds into and out of this country, as it did when the United States was on the gold standard. The gold reserve act of 1934 not only took from the system all of its gold, but in doing so definitely deprived it of future control over gold movements, although of course that power had been lost as a result of the gold embargo and subsequent monetary manipulations. With the passage of this act, therefore, the central banking system of this country formally surrendered one of the chief privileges and duties which it had exercised prior to suspension of gold payments. … The Administration has assumed responsibility for defining our monetary policies” (Washington Post February 17, 1934, 8).

所以怎么说了？美联储不给力啊，所以货币政策你别来了，让财政部给你设计就好了。美联储就像是财政部的代理人，执行财政部的政策。至此之后美联储一直没有夺回到货币政策的控制权，直到1951年的《Treasury-Fed Accord》（还是因为二战的战争债券的回报率的争端）。

上图美国大萧条时期的银行倒闭率和失业率，来自美联储主席本伯南克 Ben Bernanke在2012年在乔治华盛顿大学的课程《The Federal Reserve and the Financial Crisis》，讲了美联储的历史和使命，二战之后的历程，2008金融危机的应对和危机的余波，非常不错的课程。

因为后面金本位的问题，货币政策紧缩，没有提供足够的信贷支持，是造成大萧条那么长的一个原因.弗里德曼认为从1929到1933年，美联储没有释放信贷，增加货币供给，反而压缩信贷，导致人们没钱可花，价格下跌，收入下降，失业率增加，使得人们不得不把钱藏起来，应付更艰难的日子，进一步恶性循环。这个暂不争论，咱暂定认定本伯南克在《On Milton Friedman’s Ninetieth Birthday》说的：

“Let me end my talk by abusing slightly my status as an official representative of the Federal Reserve. I would like to say to Milton and Anna: Regarding the Great Depression, you’re right. We did it. We’re very sorry. But thanks to you, we won’t do it again.

但是为什么有了美联储，银行倒闭破产的还是有这么多了？不是有美联储提供流动性的么，实际上美联储却在相反紧缩流动性进一步加剧了银行的破产。原因目前的说法是斯特朗在1920年经济飞涨的时候实施了上面提到的欧文费雪(Irving Fisher)的价格稳定策略(price stabiltiy).欧文费雪提出应该保持一定数目的货币来保证社会商品的价格看起来稳定，比如商品产出增加，那么相应的提供对应数目的货币，让价格保持平稳。但是当斯特朗1928年死后，这个政策和低利率等等政策完全翻转（矫枉必须过正），取而代之是real bills doctrine指的货币的数目应该有其背后支撑的实际商品。结合提高利率，其目的在于打击投机倒把，降低市场热度，本意是好的。但是这样的话，货币的供给就是大幅度降低，大概下降了1/3，这种信用收缩使得更多的银行倒闭，最后9700/25000家大概是38%左右的银行暂定倒闭，直到罗斯福的银行储蓄保险才终止挤兑倒闭潮。

看的出来美联储在1929-1933年这期间的操作并没有缓解萧条情况，反而加剧了经济局面，也难怪罗斯福一把夺回了它的权利。私人银行家过的有点惨，显示投行和商行被拆分，管控越来越强，同时连本质工作都被政府抢去了。

凯恩斯深入观察到大萧条中大量的工人失业的情况，发现这跟古典经济学描述的自动均衡机制似乎不太一样，古典主义亚当斯密强调市场这只无形的手(invisible hands)会自动调节市场，不用依靠认为的力量来控制支配，应该会自动纠正市场衰退这种情况，是短期来看实际并没有，他对这种说法产生了怀疑。他认为也许市场长期来看会自动调节，走出衰退，但是问题是什么时候了？而这个中间人们却实实在在受到了衰退的痛苦，如果是一百年了？那么这一代人是不是就看不到希望了，是不是政府就站在一旁若无其事任由人民遭受痛苦了？

But this long run is a misleading guide to current affairs. In the long run we are all dead. Economists set themselves too easy, too useless a task if in tempestuous seasons then can only tell us that when the storm is long past the ocean is flat again.

（但是长远效应是一种误导。长远来看我们都死了。经济学家总是把自己看得太轻松与无用。如同风暴降临时，只能告诉大家“等风暴过去后，大海又会风平浪静的”） ——约翰梅纳德凯恩斯.

他跳出来挑战了主流的经济思想，他肯定了市场的长期自动调节机制，不过他也认为人类受不了这么长而未知的痛苦，主张政府出手多花钱创造有效需求提高总需求，帮助加快度过危机，加快经济运转，就是让劳动者们有生之年有所享受，让人心安稳，恢复社会的稳定和人们的希望，所以后面的罗斯福的新政也是受到此影响。更实际的情况，当时的情况，当资本主义陷入危机之时，苏联的发展正是如火如荼，两者形成了鲜明的对比，此时资本主义社会的政府袖手旁观不主动出手的话，内部则人民会不满，自然会倒向对面比如法西斯等；外部的同是遭受危机的其他资本主义国家有可能会选择以邻为壑的方式通过其他手段比如战争等来转移国内矛盾。这就注定了凯恩斯主义在大萧条之后的大行其道，至于其效果，有人说真正帮助美国找出大萧条的其实是二战，战争创造的需求消化过剩的产能，这个就难以评断，客观来说资本主义国家至少普遍走出了衰退（或者说推迟延缓或者说进入下一个周期）。

1971尼克松滞涨

二战后美国的马歇尔计划（The Marshall Plan），官方名称为欧洲复兴计划（European Recovery Program），是帮助美国对被战争破坏的西欧各国进行经济援助、协助重建的计划。

美国人贷款援助欧洲，欧洲在从美国买物资，美国的商品制造和出口得到了很大的好处，往外一方面输出国内过剩的产能，一方面输出过剩的外汇也让刺激了欧洲经济，帮助欧洲快速重建，提供了潜在可能人口对于消费品产品的需求力。战后欧洲实际形成了以美国为主导核心的新秩序，使得欧洲对于美国的向心力大大增强，一举两得。

1951年美联储和财政部协定之后，美联储得到渴望已久的独立性，终于可以通过提高利率来降低货币供给，希望将通胀率降下来。稍后的朝鲜战争(1950-1953)，和越南战争（1955-1975）耗费了美国大量的财力和人力。在越战美国前后期使用了不同的策略，但是总体来说越战是美国二战以来投入资金最多的战争，也付出近36万人伤亡的惨重代价。战争后期，尼克松凭借沉默的大多数（Silent Majority）赢得了两次的大选。尼克松1973年结束了越战，和中国建交，同苏联博日涅夫签署协议控制核武器，这些都没有他水门事件带来的负面影响大（1972年大选尼克松横扫赢得了51个州中的50个；另外他在水门事件中骚操作，威胁甚至一度挑战了宪法的行为 - 到现在还是有点搞不明白）。

经济方面，1968年的以通过《Civil Right Act 民权法案》出名的总统林登约翰逊，一方面扩大越战的开销，一方面搞大社会(Great Society)社会福利包括著名的向贫困开战War On Poverty和医保，进一步加大政府的开支，虽然刺激经济发展，但是通货膨胀率也跟着上升。经济的发展也使得美国进口大量的商品消费，大量用以支付的美元流出。到1968年，美国有450亿的联邦储备券的负债，但是只有100亿的黄金，也是覆盖比例大概是22%了，而美联储法案当初要求的是40%。其他国外美元持有人于是去美联储要求兑换黄金（比列还是1933年罗斯福设置的比列1ounce= $35）,导致黄金储备进一步短缺。为了让美元能相对黄金升值，只能跟当年英国一年，提高利率到6%，1969年到了9%。但是挤兑还在继续，而且短期快速提高利率，容易造成经济的震荡，出现轻微的经济衰退，失业率攀升，1970年失业率从两年前3%到了6%，通胀率回到了比较稳定水平

1971年尼克松为了总统选举，为了应对高失业率和轻度的通膨率推出Nixon Shock的经济政策：

Prosperity without war requires action on three fronts: We must create more and better jobs(创造多和好工作）; we must stop the rise in the cost of living（防止生活成本的上升）; we must protect the dollar from the attacks of international money speculators.（防止美元外汇市场投机套利）

套利的理解这里就跟97年金融危机一样，固定外汇兑换的机制本质是以国家外汇储备为基础的（这里是黄金）。以索罗斯在泰国位列，假设当时泰铢兑换美元是100：1，泰国的央行为了维持住这个固定的兑换比列，就必在外汇市场不断购买和抛售美元泰铢；如果其他国家抛售的泰铢多了，价格下来了，比如1美元对应200泰铢，那么需要用自己的美元兑换泰铢，减少泰铢的供给，让泰铢升值。金融危机前，我先从泰国银行借了1百万泰铢，去外汇市场兑换了1万美元，我欠泰国银行100W泰铢；此时如果其他人或者机构有比泰国央行更多的外汇储备，恶意的做空，不断在外汇市场抛售泰铢，使得实际泰铢的价格下跌，泰国央行为了维持固定汇率，不得不将自己储备的美元来购买多出来的泰铢；如果美元不够用，子弹不够多，泰国政府无法控制，宣布不在挂钩，停止自由兑换。泰铢也不在具有价值，也开始抛售泰铢，泰国政府被迫宣布放弃固定汇率，泰铢暴跌。这样假设泰铢暴跌，现在1百万泰铢只能兑换1美元，那么我就要那1美元去兑换然后还款给泰国银行就行了，这样我就白白得到了10000 - 1 = 9999美元。

尼克松在Nixon Ends Bretton Woods International Monetary System视频里讲解到的具体三个操作是：

控制冻结物资价格和薪资水平（freeze on all prices and wages)
进口商品加税 (imposing a 10% surcharge tax )
取消国际美元黄金的兑换 ( unilateral cancellation of the direct international convertibility of the United States dollar to gold.[)

控制物价和工资看起来很好，这里特别是指的工资和物价不能上涨。假设你现在工资不能提升，那么你的花费的购买欲也就不会提高，对于产品需求的降低，也使得企业很被动，他不能降低产品的价格来吸引消费者，还不能通过降低工资来降低成本，就只能是开除员工，这样更进一步降低了社会的总需求。一般来说控制物价只是战争时期的特殊做法，同样的这个也违背了自由市场的规则，政府在Economic Stabilization Act of 1970法案中被授权可以（总统）实施作为维护国内市场的商品和劳动力市场的统一价格项目 - 包括控制物价工资利息等等一些列价格 - 直接影响市场。

进口商品加税 - 主要是为了减少政府的财政赤字。加税的成本转嫁给出口商肯定不乐意，于是他们也提供商品的价格，是的进口产品的价格也跟着上来了，进一步增加了通胀，最后还是转嫁到了消费者身上。同时这种任意的加税也使得其他贸易伙伴的信任下降，其他人也纷纷印钱升高利率来来相对应的平衡。弗里德曼认为关税最后还是人民来买单。

取消美元黄金的兑换，实现美元和黄金的脱钩，现在美元没有任何实体价值的贵金属支撑了，全靠无形的政府的信用，成为了法定货币（legal tender/fiat money）：

法定货币（legal tender/fiat money），是指不代表实质商品或货物，发行者亦没有将货币兑现为实物义务；只依靠政府的法令使其成为合法通货的货币。法定货币的价值来自拥有者相信货币将来能维持其购买力。货币本身并无内在价值(Intrinsic value)，也就是说，当纸币产生之后，法定货币实质上就是法律规定的可以流通的纸币

之前我们看到的困难时期（比如战争萧条），国家可以暂停纸币跟黄金的兑换（脱钩），但是在困难之后还是回慢慢回到可兑换性，现在第一次出现在和平时期国家不在允许恢复兑换性。钱从此开始就是无形的，这个直接影响了后续运行的整个经济的世界的规则。

1973年尼克松彻底将美元和黄金脱钩，这个操作是的美元稍后大幅度贬值，通货膨胀率大幅度飙升，1973年8.7%，1974年12.3%，同时因为价格工资管控（虽然很快又取消了），但是已经造成了三个季度GDP负增长，失业率也上升到1974年7.2%， 1975年的9%，利率从74-75年时期始终是10%以上。火上浇油的是，1973的OPEC石油禁运(1970s energy crisis - 包括1973石油危机和1979年能源危机）造成了油价的大幅度上升，进一步让经济形势恶化。

来源:https://inflationdata.com/articles/inflation-cpi-consumer-price-index-1970-1979/

Stagflation滞胀指的就是这一时期停滞的经济增长，搞失业率和搞通胀率的统称。理论上来说弱经济和高通胀是不能同时发生的，一般说经济弱需求不振，总需求下降，那么价格就会下降，是不会发生通胀。主流观点是尼克松政府的介入加上石油危机，就是上面提到的三个操作，人为的打破了自然的循环，使得看似不可能的现象发生了：输入性通胀+价格工资控制。

这里要提到美联储的使命是什么？美联储的官方网站What economic goals does the Federal Reserve seek to achieve through its monetary policy?是这么说的：

The Federal Reserve works to promote a strong U.S. economy. Specifically, the Congress has assigned the Fed to conduct the nation’s monetary policy to support the goals of maximum employment, stable prices, and moderate long-term interest rates. When prices are stable, long-term interest rates remain at moderate levels, so the goals of price stability and moderate long-term interest rates go together. As a result, the goals of maximum employment（就业率） and stable prices（稳定价格） are often referred to as the Fed’s “dual mandate.”

失业率和稳定的价格（适度的通货膨胀）都是政府和美联储追求的双重使命，但是明显政府这边是帮倒忙了，美联储这边了？

在货币政策方面，这段时间内，美联储一直在紧缩和扩展之间不断调整利率，有通货膨胀就调高利率，紧缩萧条就调低利率，跟这阵子的财政政策和外部石油危机一样，反复无常也被称称为Stop-Go Monetary Policy，普遍认为这样的货币政策使得商业投资变得摸不着头脑（对于商业来说很重要的是稳定的预期和政策），所以反而加剧了滞涨的情况。1960年代为了最大化就业，美联储一直保持这低利率，是的大量的货币进入流通；基于此，弗里德曼分析之后第一个用货币理论成功预测了1970年代的滞涨现象，他认为通胀是政府收取的另外一种税，这种税只跟货币供给有关，跟工会或是不同国家情况没有关系1978: Milton Friedman Speaks: Money and Inflation (B1230) - Full Video。他简单描述了什么是通货膨胀：

Inflation is Too much money chasing too few goods. （通货膨胀就是太多钱追着太少的商品）

弗里德曼认为美联储这阵子的问题就是货币供给的不稳定，所以他认为美联储应该像狗一样用链子锁起来，或者应该说只一个没得感情的计算机程序都比美联储这群人靠谱，每年稳定的往市场供给一定比列的货币就可以了比如3%。

从大萧条和二战危机中帮助政府渡过（或者说推迟）来的凯恩斯主义，认为政府应该扩大开支增加投资，主动弥补缺失的社会总需求，创造就业，来刺激经济，促进经济的增长。简单描述下在凯恩斯之前的古典经济学派认为价格和工资都是富有弹性的，所以价格和工资的自我调整机制可以促使市场迅速恢复平衡。

例如：当大量工人失业时，企业可以降低工资水平以雇佣更多的工人；反过来，雇佣量的提高又促进了失业率的降低。当大量工人就业时

所以说古典学派认为非自愿的失业是因为那些人不愿意接受低工资才会，由于工资的弹性调节机制可以确保失业人口很快地重新找到工作，非自愿失业是不可能存在的。另外,即使工资水平较低，人们的消费需求也会刺激失业人口接收这份工作的。

这套理论在平常可能没有问题，但是凯恩斯在大萧条的发现大量的人失业，这个已经是古典学派“不存在非自愿失业”的论点无法解释的。他认为在在经济萧条情况下，即使大量工人失业，企业也不一定会用低工资招人，因为需求没了，你生产出来商品没有人要，价格再低也没用，因为消费者的钱都藏起来了，可能明天的价格更低，不如有钱在手（巴菲特还说在危机时刻掌握现金最保险了 - 也就是有效需求降低，有钱但是不愿意消费）；另外此时银行即使有钱也不会轻易放贷出去，进一步恶化情况；

这种情况自由放任的经济就陷入困境，如果不介入，无形的手说不定可以慢慢恢复但是政治成本太高（有争论）；凯恩斯认为政府应该增加投资支出来恢复经济，比如08年我们的财政部的4万亿财政计划，他说财政部可以把钱装在瓦罐里，埋在废弃的煤矿里，让资本家雇人去挖就可以启动经济,从而提供就业和收入.

简单的说：凯恩斯主义认为失业与通货膨胀是成反比的。比如通胀是因为低失业率，也就是有效需要高，需要大于供给，当经济好时，人们愿意消费，使得物资价格上升；当出现高失业率时，需求小于供给，导致价格下跌，也就是少消费的通过紧缩；这个关系反映在菲利普斯曲线上。而货币主义认为通货膨胀从根本上讲是一种货币现象。

凯恩斯主义的失效，标注着货币主义的兴起。打个比方，如果说经济发展像是开车的话，之前只有凯恩斯主义的一副油门和一副刹车，但是目前这个失效了，车控制不住了；怎么办了？那就试试再加一份油门和刹车-货币主义。

自此货币主义开始占据主流，1979年卡特认命保罗·沃尔克(Paul Volcker)为美联储主席，开始将货币主义的理论付诸于实际。

[2020年关于现代经济的一些思考(三)]

2020年关于现代经济的一些思考(一)

2021-02-01T14:55:32+00:00

刚刚过去的2020年应该说是灾难危机的一年，非常特别的一年，当一月份还在为科比意外去世惋惜时，新冠病毒，黑人运动BLM,，种族歧视，美国大选还有三月份的股市连续崩盘熔断，经济放缓，失业率暴增，随后的美联储救市，财政刺激，在美国新冠死亡人数不断创新高的情况，股市一路高歌猛进，中间也有黄金的暴涨，到了年尾有比特币的暴涨，贫富差距加大。大选上，特朗普在普遍媒体民调不看好的情况（甚至90%上会败给拜登 )，结果出其意料的跟拜登不相上下，后面一系列指控选举舞弊（Stop the steal），甚至在今年1月6号发生了支持者冲突国会的暴力事件，拜登斥责某些共和党人像是纳粹的宣传部长，而特朗普也被社交媒体禁言 - 党派分裂的趋势越来越强。作为程序员可能尽管个人跟经济打的交道不多，但是每天都能接触到很多经济方面的术语”放水“、”比特币“、”股市“、”熔断“、”黄金“、”美联储”、“央行”、“通胀”等等，这些社会热点被动输入到我的大脑中，很多时候就是一看而仅此而已。之前难得有空在老家待了一阵，突然想起来之前一直狂轰乱炸的那些文字概念，回头一想，可能是一个学习了解新事物-货币和经济运行机制-的好机会。毕竟我们每天都跟钱打交道，却是很少去思考现象背后的运行的规则是怎么样的，这是个好机会重新挖掘一下自己的已有的知识，有哪些持有的印象概念是对的，有哪些肤浅的，整理归纳顺便学习新的知识，应该是蛮有意思的。我个人不是经济专业的，没有系统学过经济方面的知识，文章里写下的就是我私下觉得有趣然后花了点时间慢慢看学习，结合自己的实际观察，吸收总结归纳的一些见解，应该是有很多疏漏，可以指正的。

截止到2021年2月3号，根据https://covidtracking.com/美国确诊的新冠病毒病例有二千六百万人，死人总人数四十四万。

我们经常听到了一些看人们常说的说法和观点比如股市是经济的晴雨表、当比特币暴涨的时候很多人说比特币涨的原因就是美联储放水、当比特币跌的时候人们说比特币的价值本来就是0，量子计算机的超强算力就会轻易的破解加密算法使得其价值归0、美国股市一定是处于大的泡沫随时可能崩塌、美联储是私人银行家的掌控的是一个巨大的阴谋等等吧，这些说法和观点一直困扰我，虽然我无法肯定是否对错。但是这些观点和看法使我立马想到了黑格尔哲学里辩证法说的静止和抽象。以我浅薄的理解，黑格尔的“合理的内核”中的辩证法认为是实在真理（他认为的绝对精神Geist）是不断运动、变化、发展的，每个环节每一个方面都不是固定不动相互绝缘的，他们之间存在着内在的联系和转换。”具体“真理也就是是多样性的有机统一的，包含着事物不同方面有对立也有矛盾（抽象），因此它们不是静止和固定不动的，而是能互相转化和矛盾发展。”具体“指的是多样性的有机整体，”抽象”是是隔离的片面的局部。比如花如果是一个整体的话（具体），它具有很多特质但是它不是这些特质的堆叠，比如颜色，气味，形状等等。而花的某一个性质比如“颜色”，就是从整体中抽出来的，使得跟其他特质撕裂开来，那么颜色就是一个“抽象”。黑格尔认为世界上的事物都是具体的，都是许多方面、因素或性质的内在联系着的统一体，无论在天上或地下，无论在精神界或自然界，绝没有“抽象的”、孤立的事物；任何事物如果把它绝对孤立起来，那是毫无意义的。

所以联系起来就是这些描述让我感觉只是描述一个事物的某一个方面和某一个时间的静止的状态，自然的就会反问到会不会尤其没有其他方面，它的历史是怎么样的，怎么演变发展到这个阶段的，将来会是怎么样了。这种目前的-特别是不符合常理的-现象体现出来的矛盾性是不是能体现其本身内在这个阶段的本质，这个论断是不是有道理有逻辑支持，还是只是简单情绪的表达和发泄。不管怎么样，这都值得让人“叮咚”一下，诶，提醒自己保持好奇可以尝试验比较下自己的已有的印象和认知。

关于美联储今日的问题，某项经济制度和措施可能目前来看是有问题的，但是如果不仔细研究其历史和背景，不得出具体的分析和逻辑思辨，简单的认为只要推到就可以解决一时之问题，先不论是不是能解决当前问题，还要看看说这话之人是否可以拿出切实详实的新秩序的方案，否则只是一时之愤，可能甚至被利用或是悔不当初。就像这位老哥一样“I toppled Saddam’s statue – now I want him back” BBC News.

另外正如钱穆先生在《中国历代政治得失》中讲的：

某一项制度之逐渐创始而臻于成熟，在当时必有种种人事需要，逐渐在酝酿，又必有种种用意，来创设此制度。这些，在当时也未必尽为人所知，一到后世则更少人知道。但任何一制度之创立，必然有其外在的需要，必然有其内在的用意，则是断无可疑的。纵然事过境迁，后代人都不了解了；即其在当时，也不能尽人了解得；但到底这不是一秘密。在当时，乃至在不远的后代，仍然有人知道得该项制度之外在需要与内在用意，有记载在历史上。这是我们讨论该项制度所必须注意的材料。否则时代已变，制度已不存在，单凭异代人主观的意见和悬空的推论，决不能恰切符合该项制度在当时实际的需要和真确的用意。

一项制度可能发展久了到慢慢僵化了现在看来有问题就一味批判就一无所用全盘否定是不合适的；也不能用现在的观点和场景去套用在过去的事物上，当时事物有当时的发展形势和出发理念，只有学习借鉴批判才能更好的应对当下，时代意见不能抹杀历史意见。

秉承这样的思路，我梳理了下钱和美联储的历史，然后分析了下当前的形势。

简要英国货币史

先从英国开始吧。如果圈子比较小，以物易物是可以的，特别是危机战争等时候，人们还是会选择原始的以物易物；但是圈子比较大时候，贸易交换越来越频繁时，这时候就需要一种合适的媒介来保证携带的方便性，一个是媒介本身的物理价值稳定大家都认可，这个时候贵金属就成为了主要的货币了。金银铜，这种依靠成色的货币本质上还是依靠金属本身的价值作为等价交换物而存在。从以物易物的原始社会到重农抑商的封建社会后，货币体系慢慢形成，比如古罗马的货币体系就是1金币=25银=400铜。铜的额度太小，价值不高携带起来不方便，所以只能是配角；主要是还是银和金。但是因为古罗马一直战争，需要举债，但是货币供给量实在不足，于是统治者要么减轻重量要么掺假使得劣币超发，成色好的货币都被贮存了起来，甚至不得不退回到了以物易物，造成了全面的衰退。既然市场上流通的都是劣币，没人愿意使用，戴克里不得不宣布重新铸币来替换掉旧币。地理大发现之后，随着贸易量的上升和西班牙在美洲秘鲁发现超大银矿，使得大量的白银流入欧洲。

自1158年以来英国一直流行的银为主的标准，铸币也是按照金银含量大小的不同分为了不同的单位，类似1英镑=20先令=240便士。

比如英国国王詹姆斯二世时期的“基尼”金币Guinea铸造于1685-1688期间，重8.4克，直径25-26mm,纯度大概在0.9094，一英镑。正面是国王的头像，然后四周被环绕的字是 iacobvs ii dei gratia (“James II by the grace of God”)詹姆斯二世承自上帝的荣耀，反面是十字形状的带皇冠的盾牌。基尼是因为金来自非洲的产金国几内亚（Guinea）。这样看这纹理和文字工艺复杂度(有兴趣可以看看现在人民币如何防伪回形针 - Vol.099 造假币为什么这么难？，以当时的条件，普通人要熔化造假也挺难的，难怪后面不断的重新铸币发币了。这些是英国历史上发行的金银币，可以看看硬币上的花纹雕刻和图案：Coin Gallery

17世纪英国人将金银等贵金属存放到金匠铺（为什么是金匠而不是国家官方某个地方了？查理一世威胁苏格兰改成新教仪式，想发动战争但是议会拒绝征税，他强行霸占控制了当时英国人普遍存金子的地方”伦敦塔”-牛顿后来担任的皇家铸币厂所在地，结果自然人们都纷纷转而存到私人的金铺里），那里时会收到到金匠给的一张类似存款仓库收据，这张收据可以被转让，也可以由储户凭该收据去金铺兑换金银币，金铺收取一定手续费保管费。本票（Promissory notes）也就出在这个，大概意思是：发票人谁谁有固定数目的金条（多少钱）给到谁谁持票人（Bearer），然后拥有人可以通过签字将单转让给其他人，只要上面还有空间就可以无限转手。 (I promise to pay the bearer on demand the sum of... 跟当前我们的借条开头一样的).本票来源：

私人合同，是没有法律效应的，纯靠对金铺的信任，所以如果金铺主人财务状况好，守诚信，那么他更容易收到储户的信任。但是私人合同出现了法律纠纷时候怎么办？谁来做调停人？这些问题迫使议会通过了1704年的《PROMISSORY NOTES ACT OF 1704 本票法案》:

“An Act for giving like Remedy upon Promissory Notes, as is now used upon Bills of Exchange, and for the better Payment of Inland Bills of Exchange.”

政府做了立法上的支持，本票可以作为可支付的载体，对于持票人和发票人拥有对应的法定责任，票据可以转让给新的持票人，并且受到法律保护可以在适当时候起诉来强制执行赎回。这个给了当时的日益增长的商业和贸易提供很大的便利。

本票正是银行券Banknote- 是由银行（尤指中央银行）发行的一种票据，俗称钞票 - 的前身。上面的金匠的票据进一步发展去掉持票人的名字Bearer去掉，然后将金额固定住，这就是向钞票的进化了。BTW，写代码的http请求很多需要写Authroization的头部字段，其中有一种就是Bearer Token，或则现在用到的认证方式JWT(Json Web Token)主要是以Bearer的方式校验。这样联系一看很合理了，持票人持有这个口令。稍后半票改进下加入了一些防伪的雕刻就更进一步能转化为钞票了。

下面是英格兰银行发行的第一张5英镑的钞票,两面的。（来自英格兰银行官方历史时间线 - The first £5 note - Bank of England）

插一句话就是美国各个殖民地也都发行自己的纸钞，形成了各具特色的设计方案，这篇文章《The Ornate Charm of American Currency from the 1700s》详细介绍了不同种的钞票风格，还挺有美感的。下面是特拉华州殖民地的一张4先令的背面(A four shilling note from Colonial Delaware - back side):

可以看到各种各样的雕刻方案和花纹，同时也有的写着 “To Counterfeit, is Death."来恐吓下面胆敢造假币者。后面独立战争期间大陆联盟（Continental Congress）也发行自己的正式的大陆联盟币。下图是3美元的大陆币，画的跟鬼画符似得A three dollar Continental bill. (图片来自: National Numismatic Collection/Public Domain)

跟历史上的战争一样，因为独立战争举债，Continental currency超发贬值。战争之后，大陆币贬值如此厉害（有了Worthless Continentals垃圾大陆币一说），以至于人们拒绝接受大陆币来购买商品或贸易。贬值导致的物价飞涨和需要支付大量的战争债务，为了稳定经济和解决政府债务问题，汉密尔顿提出了建立一个国家银行的想法。也直接导致了第一美国银行的建立，同时为了恢复国家信用和清除劣币，跟历史一样，有劣币那就重新发新货币，同时通过了1792《铸币法案》规定银元是衡量其他货币的基础，银和金的比列是15：1，实行金银复本位制，将用硬币替换掉流通的纸币直到美国内战的爆发前。

1694年光荣革命之后为了建造更多的战舰加强海上力量和法国抗衡，英王迫切需要资金流，于是创立了英格兰银行(Bank of England) -宗旨是：promote the public Good and Benefit of our People。它是一家私人的银行，股东持有股份，帮助政府融资，政府发布特许证(Royal Charter of 1694)背书，当然英王也是原始股东,来发行政府债券式的钞票(Banknote against the government bonds)来向公众融资借钱。民众借钱给政府钱，得到发型的票据(note)，上面注明利息(%8每年)，这个票据可以被转让，通过这个方式政府很快在12天里筹集到了一百多万的英镑，一般用来建造海军，也奠定后续英国的海上霸权地位。

这个时候如何衡量一个国家是否强大了，这个时候主要还是人口数量和土地疆域。1800年前的欧洲社会正如马尔萨斯观察到的人口和土地的矛盾是不可调和的，要么你有更多人口，更多竞争，而土地生产资料不足，生产关系在工业革命之前普遍没有啥进展的情况下，那么只能是通过宗教战争瘟疫等等来平衡人口和土地的关系。法国是欧洲传统的大陆强国大国，英国作为岛国小国，土地面积狭小，且不经过海洋的话无法拓展，只能是跟法国竞争来获取更多的海外殖民地，让海外殖民地来养本土，降低本土的国内矛盾。

政府将借到的钱投资在建设海军军舰，由于这些巨大的工业需求，那么进一步带动了比如建立钢铁厂的锻造钉子等，农业技术进步来给更多人补给吃的等等带动了很多部门。而这些各部门各领域交换中中具有轻薄便于携带同时有法律支持的特性的银行券提供了巨大的支持。

政府借出去的钱总是要还的，主要是通过一方面通过一般的税收，一方面战争获得的赔付，还有批复一些特许状获得部分股权利益。政府可能还不上，但是政府的公共属性和信用度还是比这些金铺私人的靠谱的，所以更多人愿意买政府的债券。

与此同时，多年来以银为标准的本位制也发生了变化，1696年牛顿担任皇家铸币厂的总监。当时一个是白银在使用过程中存在的耗损严重等问题，另外劣币假币充斥、良币被贮存了起来，这些情况牛顿想扭转重新确定以白银为本位的新币，他打击私自销熔银币、伪造劣币等，同时还需要从民间收集白银来融化铸造新币。还有一个突出问题是英国的金银价格比列跟欧洲大陆国家的不一致，导致了严重的套利。当时金银有一个兑换比，英国比其他欧洲的都要高。

比如上面极端假设情况，英国1黄金对10白银，法国1黄对1银。我在英国用1盎司黄金还的10盎司银子，然后去法国将10盎司换成10黄金，在运回英国，那么我就赚了到10-1=9黄金的差价。英国黄金越来越多，白银流失；法国白银越来也多，黄金流失；这就使得英国国内白银币质量越来越差，整体社会中的货币供给量减少，使得人们转而到了质量更好的”基尼”金币上。牛顿很快意识到了英国金银兑换比率的问题，同时白银国内没有足量供铸造用了，因而他建议将金银兑换跟欧洲接轨。

在1717年牛顿在《向上议院财税委员会阁下的陈述Representation to the Right Honourable the Lords Commissioners of His Majesty’s Revenue》中说道：

“自基尼（金币）被铸造以来，就一直处于被高估的状态，在以银币为本位货币下的英格兰，基尼的被高估导致金银兑换比失衡。而在欧洲大陆各国，银币的价值较高，这就造成我国国内的银币外流，而国外的金币不断流向英格兰……因此，将1基尼定为21先令Shilling才是合理的。”

英国议会认可的牛顿的提议，牛顿的报告也让英国实质性地进入金币位制时代，这一年成为货币史上一个重要的事件。

回到纸币上，1725年英格兰银行开始发行固定面值的银行券。可以根据这个票上的数目从银行取出对应面值的贵金属硬币的。所以换句话说，在电子货币出现之前，这张票据（纸钞）实际是被具体的实物-具有物理特性的价值载体-贵金属-担保的，以黄金为本位。银行必须确保它自己有足够的黄金储备才行，否则票据就无法兑现。。1770年代后，由于支票的流行，大多数伦敦的私人银行都停止了发行银行券，而使用英格兰银行券，需要时再到英格兰银行兑取黄金，英格兰银行券也因此成为了金币的代用品。到1776年亚当斯密在发布的《国富论》中观察到流通中的银行券实际已经超过了实体硬币，也成为了货比历史上一个转折点。

1759年的英法七年战争和稍后1793年英法关于殖民地的争夺，冲突升级可能1797年本土遭受战争的消息，恐慌导致人民纷纷去银行兑换金银币，一些私人银行挤兑倒闭，黄金流失。英格兰银行宣布暂停银行券的兑现金币Bank Restriction Act 1797从1797到1821期间禁止银行兑换金币，有点像罗斯福1933的黄金法案里宣布的bank holidays银行假日来禁止群众美元兑换黄金。1816年，英国通过了《金本位制度法案》，以法律的形式确认了黄金作为货币本位来发行纸币的地位。1833年，英国议会通过一项新法案，规定英格兰银行券获得“无限法偿”资格（即法定货币，强制流通，必须接受，不得拒收）。英格兰银行券成为首个、也是唯一获得法定货币地位的银行券，在法律上地位等同黄金（1816年确立金本位）。而其他银行券也可流通，但都不是法定货币（非强制流通，人们可拒收）。

1844年7月，英国议会通过《银行特许法案》（The Bank Charter Act 1844）确立了只有英格兰银行才行纸币发行权的地位，其他银行后续慢慢退出了。1928年，英国议会通过《通货和钞票法案》（Currency and Banknotes Act 1928），在法律上确认了英格兰银行为全国的货币发行唯一机构。1931年英国彻底脱离金本位后，英格兰银行券成为了“金本位接班人”，成为了英国的法定信用纸币（仅靠政府信用背书，不可兑现），并延续使用至今。英国货币制度也转变为了不可兑现的信用纸币制度。英格兰银行成为了现代意义上的中央银行。

19世纪会出现的“大分流”，随着英国工业革命和强大的军事经济实力，大三角贸易和工业革命分工将日不落帝国的所有殖民地或主动或被动的拉倒了这个统一的市场里，同时其他落后的国家比如刚刚由俾斯曼统一的德国1870年加入金本位，新生的美国等等，都不得不得跟英国有贸易往来用英镑结算。他们的工业发展要随后跟上，就不得不搭上英国这辆车，就必须接受英国的制度和规则，英镑成为了第一个全球性的货币，奠定了英镑的霸主地位。英国的科技（火车蒸汽），经济（利物浦是金融中心），军事（海上的绝对力量），道义（虽然英国不是第一个贩卖奴隶的，但是确实贩卖的最多的，却也是第一个实现废奴的国家）实力成为了英镑背后的支撑。此外英国禁止殖民地擅自跟非英盟的国家贸易一英国就相当于今天美国。

金本位的好处是每个国家都可以将自己国家的货币设计跟黄金的兑换比例，然后只要换算一下就可以得到跟英镑的数目，非常稳定固定的的货币体系，极大方便了当时日益增长的贸易交换和国际投资等等。

有意思的是，理论上和常识是上这些钞票是跟黄金对应的-但实际并没有100%覆盖，也就是不是一张纸钞会有对应量的黄金。实际上1913年的美国美联储法律要求的是40%覆盖即期票据，而1910年英国是46%，德国是54%，也并不是真正的1比1对应关系。

简要美国货币史

1800年后随着美国西部加利福尼亚金矿的掘金潮（旧金山）和世界其他地方产量的增加，使得黄金价格下跌开始逐步替换了以银为主的本位制度。1861年内战时候，两边巨额的战争开支使得政府放弃了传统锚定货币（在这之前美国直有硬币，金银币；私人银行可以发行银行券，但是那不是法币（legal tender-Legal tender is a medium of payment recognized by a legal system to be valid for meeting a financial obligation). 如果银行垮了，就没啥用；国家发行的比如过去的大陆币纸币是法定货币，但是只是假装在某个时间可以换成金银币，实际上是没有实际的金银的支撑的，大部分情况是空头支票，所以人们很快知道了这个小伎俩，贬值的特别快，改发FIAT(无锚货币)纸币，也就是没有贵金属支持的货币 - 仅仅靠的是政府的信用。

政府收入来源大部分是税收，关税和消费税；但是这个时候明显战争的开销明确超过了实际可印制的钱（来自税），绿皮币 - Greenback

绿皮币没有利息，但是可以兑换为金银币（on demand见票即付），而且可以用来支付关税。但是很快就发现没有足够的金银来兑换绿皮纸钞，所以1861年政府暂停了兑换，随后拒绝了硬币兑换。事实上1861年财政部花费了2350万美元，但是只有580万收入，入不敷出；好在内战林肯赢了，绿皮币的价格得到了回升，但是货币超发相对金币贬值还是不争的事实，如果要回到金本位就得减少货币印刷，但是要会到重建产生负面影响，通货紧缩，失业率增加；只能是慢慢减少发币(同时减少其他货币流通，1873年的《Coinage Act of 1873》停止铸造银币，废弃了流通中银币，也被称为Crime of ‘73），同时通过投资铁路等等刺激经济发展，得到更多的经济产出，从而提升货币的购买力（新增钱多生产货品少->新增钱少生产货品多)，直到近1880年才回到金本位，这里大概花了15年才回到当初的购买力水平。这个中间需要一个慢慢尝试的过程，而且是内战(外部条件几乎无变化)；相比后来英国一战后立马回到金本位的做法，急于求成了.

[2020年关于现代经济的一些思考(二)]

An AWS Serverless Experience

2020-06-06T14:55:32+00:00

Background

A couple of months ago, we got a project which need implement sth like IoT like running or jogging apps but slightly different. Basically the backend admin could design specfic route for tasks(like delivery service), users could compete for the tasks to get bonus. Once user got the task from the app, he could start the task according to the route info and upload its coordinates like every 10 seconds. Then the manager in dashboard could live monitoring echo ones’s progress on the map. Nothing too complicated, plus user could take photo of some receipt(like invoice) , upload it and backend should be able to do some OCR to extract correct amount of money. Every monthly or weekly, some reports should generated for analysis.

Technical demands from client:

sort IoT style project; (DynamoDB for gps)
need use Python or Java for language stack; 
Serverless structure based on AWS infrasture (Favoring AWS native techstack)
has to work in China (e.g. use AWS CN)

For second point, i.e, language part, given that there isn’t too much about transaction, and Java is obviously heavier comparing to Python, so Python looks like no brainer to us. There is a good video from AWS Reinvent 2019 on Java on AWS serverless: AWS re:Invent 2019: [REPEAT 1] Best practices for AWS Lambda and Java (SVS403-R1).

Framework Options

Initially we start with raw AWS tech stack to get familiar little bit (As none of us have any experience with Serverless, only some Docker and Kubernetes before), playing IAM, APIGateway, Lambda, CloudFormation, DynamoDB and RDS (the database is set to use Postgresql) with online editor. Not too bad for getting our feet wet. Once we got the concept of those, we found that The AWS Serverless Application Model (SAM) looks like a decent way for developers to work with. We could weave infrasture-as-code(simplified CloudFormation) templates with actual bussiness code (python here), streamlining the development and deploy process.

Still you need write quite lots of configuration code, even though your main focus should be on actual business code. After some investigation, we could Serverless Framework/AWS Chalice/Zapper/Claudia.js)

Now obvisouly the Serverless Framework is most popular one, it even works with Aliyun(Chinese version of AWS), and it really extract out lots of low-level details and help developer to focus on business logic development. But client reject this as they dont’ think it is AWS native enough to make best out of AWS’s potentials :( Maybe? There are some key differences between AWS international and AWS China which mightn’t work in China Region if we dig deeper later.

AWS Chalice looks promising as it is coming out of AWS’s own hands.

Neat, nice and super developer-freindly. But here is imporant part, AWS and AWS China are different. Like some discussions here and there on github: Add support for China region #792 , Using serverless framework for deployment in China region, Bring basic support for AWS China #1564

some are already or could be resolved or supported by AWS China like:

environment varaibles (EnvironmentVariablesFeature) - later supported
EDGE API gateway to set to REGIONAL
403 - Ensure your domain is ICP listed (or you just want to play with it now, ensure you explicitly open a ticket in AWS China panel asking about open temporay API Gateway permission)

The biggest one is that ${AWS:Partition} is different according to AWS ARN and namespace. Also there some services are not supported in China region, like AWS AppSync.

some resources, s3 or iam, for example, arn:aws:s3:::my_corporate_bucket/*, would be arn:aws-cn:s3:::my_corporate_bucket/*. Also not every AWS service are supported in China Region: aws china region product services listS, for example, one key part of CI/CD CodePipeline is missing in China region, but CodeBuild and CodeDeploy is supported, so later we need replace it with Jenkins (and help of some plugins) to customize CI/CD flow.

In conclusion, we choose SAM and plain python(no web framework here) for develop backend.

Module Breakdown

In microservice specially, with some reminsicence of old day Java OO programming, Single Responsibility Principle always caught me. Robert.C.Martin, long time ago in his book《Agile software development principles, patterns, and practices》, mentioned five OO principles, and first is the SRP/Separate of concerns. This applies to Class/Function etc, not for code perfectionist, as it stands for good abstraction, better cognition and easy understanding. Each lambda in AWS represent a business unit, and it should and just do what it needs to do, nothing more or less. Like superstitious/obscure theology, followed by Age of Reason(btw, salute to George Carlin) and Englightment, everything should be based on its logic, but even emotions and human affections? lets pause here for Romanticism, no doctrine will be silver blullet as context always evovles.

In serverless world, we need consider here:

Single Responsibility Principle and decomposition - easy test, lightweight, faster startup time (cold)
lifecycle/duration/frequences - there could one off or long time running
event source - how does this got triggered ? (from s3, or request or ….)
resources/permissions applied
pricing

Development

Code structure

There are many ways to organzie the code structure. For example, for CRUD case, you could have this:

team
├── create
│   ├── app.py
│   └── requirements.txt
├── get
│   ├── app.py
│   └── requirements.txt
├── list
│   ├── app.py
│   └── requirements.txt
└── update
    ├── app.py
    └── requirements.txt

or you could put CRUD in one app.py like old days:

team
├── app.py
├── requirements.txt

and in template.yaml you could declare the function:

When you run sam build --debug, from the logs:

from official website on sam build:

The sam build command processes your AWS SAM template file, application code, and any applicable language-specific files and dependencies, and copies build artifacts in the format and location expected by subsequent steps in your workflow. You specify dependencies in a manifest file that you include in your application, such as requirements.txt for Python functions, or package.json for Node.js functions.

So it loops each folder then run pip install there, and copy the source code and installed packages to .aws-sam/build folder.

Then when later sam package/deploy, it will zip the whole folder AdminAuthFunction and upload to somehere in s3. Once the aws try to execute lambda, it need do some warming up, setup basics, pull the zipped code from s3, unzip it and etc.If the code zipped size is smaller, it would means faster cold start (downloading faster) and less code mess, as it only contains what it needs.

Personally I favor the second one, as later I would mention about the swagger definitions, - which I think swagger(API) , template yaml(Resource), app.py(Code) and requirements.txt(Dependencies) should be one single unit as it represent one self-contained bussiness logic. (This is some cons of the current aws stack, I will elaborate it later)

To share code inside different services, we could use language specfic solution like npm or pip. Here multiple services share some common logic like utils are pretty common cases. One thing about use pip/npm package, is that it is not easy or nature to integrate with part of development process. Also each service need download the dependencies separately when cold startup.

AWS Lambda layers are pretty useful here.

You can configure your Lambda function to pull in additional code and content in the form of layers. A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package.

Working with AWS Lambda and Lambda Layers in AWS SAM give pretty good example on how it use with SAM to be part of dev process by directly declare and use it in your sam templates.yaml.

One good use of layer is to lock down system-wide built-in dependencies explicitly, like bot3.

Different environments

There are some good SAM best practices from AWS talks and docs. Some are quite practical ones:

ideally have seperate AWS accounts for different dev/uat/prod environments
use AWS System Manger - AWS Systems Manager Parameter Store for keys or secrets/credentails(e.g password) rather than plain hard-coded in code.
APIGateway Stage Variables
Use Parameters and Mappings

Test and Deploy

Local Dev/Test

After sam build, you could run sam local start-api to start locally to debug and test it.

Mounting CreateOrderFunction at http://127.0.0.1:3000/orders [POST, OPTIONS]
Mounting ListCarFunction at http://127.0.0.1:3000/cars [GET, OPTIONS]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-06-06 18:51:23  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)

on my mac, SAM CLI, version 0.41.0, I got interested on this line:

You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template

But truth is, when I just change the code, and call via curl again, nothing happened. But as I guess, it is when you actually do curl http://127.0.0.1:3000/orders , only then SAM will try to invoke the lambda, download docker images (runtime, layer etc), and mount

Mounting /Users/tuo/Documents/6e/xxx/backend/route-api/.aws-sam/build/ListOrderFunction as /var/task:ro,delegated inside runtime container

from .aws-sam/build to docker container’s volume. So either it means sam build has some magic watch command ? or there is an option of watch in sam build command? But nope, I didn’t find any like that in manual.

So what you could do is just simply go to .aws-sam/build folder , find the correct function directory and directly modify from it. But you’re gonna remember to add it back to source code when done, or if you run sam build accidentally (I guarantee you that it will, as I did that a couple of times and lost all codes/changes even if jetbrain smartest IDE could’nt get it back from my GOAT feature, i.e, Local History: in IntelliJ IDEA May Save Your Life Code). No git stash or git commit or what so ever, because it often got git ignored! Hoorooooooay !

I will talk a little bit improvement later for this process. But lets continue for deploy part.

Deploy

After you run sam build, you probably notice in terimal, you could do sam deploy --guided. But if you like me, are in China, you know it won’t - same arn parition problem.

So you need:

create s3 sbucket for deploy (only once if bucket doesn’t exist)
```
aws s3 mb s3://tuo-serverless-artifact
```

sam upload package

sam package --output-template-file packaged.yaml --s3-bucket tuo-serverless-artifact

sam deploy package

sam deploy --template-file packaged.yaml --region cn-north-1 --capabilities CAPABILITY_IAM --stack-name demo-tuo-api  --s3-bucket tuo-serverless-artifact --confirm-changeset

But how about you need have some decent CI/CD for automating deploy process?

Unfortunately, the codepipeline is not supported in China, we need use Jenkins/TravisCI for replacement. You could use the jenkins plugin AWS CodeBuild with github to setup code trigger to codebuild process.

CodeBuild basically do sam build and package, after that the zip and artifact should be upload to s3 ready for aws cloudformation to deploy.

Here is a pretty tutorial on Jenkins with CodeBuild&CodeDeploy from aws blog: Setting up a CI/CD pipeline by integrating Jenkins with AWS CodeBuild and AWS CodeDeploy. My experience with this is that this process is very long and error-prone, sometimes you need step back and think why and it might not fit perfectly on your case, and twist little bit.

Drawbacks and Limitations

let’s review what are some limitations from aws serverless and SAM.

`sam build` is slow

There is some issue on github: “sam build” feedback: Support incremental builds #805.

One of reason, as I mentioned earilier, it is need loop folder, and run pip install requiments.txt everytime, even if you didn’t change it.And I didn’t see it uses any cache for quick install. (I did that by time sam build command for clean install and later install, and pretty much no differences. plus we could use –debug to see details).

One way would be to use sam build help to list all functions and only run functions you have changes and want to build for that specific service like sam build AdminAuthFunction. But this would add cognition overhead, as we need shift our mindset from for example: team -> create -> app.py to CreateTeamFunction in template.yaml(see reverse order). I guess you could make naming and folder structure consistent by for example, function name in templates.yaml should be {parent}capitialize{sub} like team->create folder to teamCreate function name in templates.yaml, whicy is way easier when you need navigate from aws console’s lambda panel to locate function.

if you are in China, you could run change pip source to use Chinese package index. Plus you could do quick smoke test directly inside .aws-sam/build folder.

But any good way to help this process? One propose would be:

Loop each folder, check if reuqiments.txt is changed or not(save as md5 signaure and compare with it ).
If this md5 doesn’t match new one, then pip install and copy packages to build folder, and copy the all code inside directory
If md5 same, just copy the source code

Copying source code is pretty lightweight and fast. I haven’t got some time working on this but I guess Gulp could be good tool like what we did old days to watch scss/js changes.

sam local start-api is slow

One good way to speed up is to add --skip-pull-image to the command to skip re-download if you have it already.

But the most painful part is it need start new container and mount volume etc everytime you request it. There is one ticket Feature request: make it possible to keep docker container warm but author said aws team is review and prioritize this task. I guess in short term, it wouldn’t be ready.

Okay, so sam build one single function(empty requirement.txt, no dependency at all, but stil it need run pip install) takes 1.0 seconds and an request to it takes like in above picture 13~ seconds even thought it is the second time requesting.

And there is not watch for sam build, you need manually switch back and forth between terminals panels and typing different commands.

Can’t run custom authorizer locally with API Gateway

Feature Request: API Gateway Authorizer support in SAM Local #137.

We need add JWT for custom authorizer with cors support. But we can’t test this locally somehow. I understand the whole thing, docker etc, aws team try to minick the real production environment with consistency to help migitate the pains of the this-thing-doesn’t-work-on-my-laptop. It did! But obvisouly something still need to work on more.

separation of concerns

let’s give a quick glance of template.yaml I got from other project which given as a sample or refeference for my team/project to refer to. (live in production, not some random/fake ones from internet). I did learn a lot from those references, but I found it has some problems.

For APIGateway yamls, which mainly containes parameters and mappings, swagger api definition and other resources, has over 3 thousands lines. That’s pretty insane! Imagine you need change some part of code here, it would be undigestible and super hard to locate and change it. Just imagine again, you need change some swagger mapping template in velocity language in a yaml file.

After a quick look, I found the swagger api definition part takes the most. Again, my two months development experiences from novice tells me that there is some thing could be improved.

The thing is that I need constantly jump between python code and template.yaml to change both, which is kinda really painful. The python code containers source code and dependecies, and template.yaml containes instrastructure related and api definitions. Remember, take for example, rails or expressjs, you have controllers/services/models/routes in separate folders, meaning you have user model, you need have user-related code in 4 or even folders, well what they do is just simply cooperate to do one sth - user.

So as I mentioned previously, it would be better that it wraps around the business logic.

list
├── app.py
├── requirements.txt
├── swagger.yaml
└── template.yaml

Each service or business logic has source code, dependency, swagger api and template.yaml for IaC. But too fine-grained level would simply introduce more botherings, also there are some real-world limitations we need consider.

team
├── create
│   ├── app.py
│   └── requirements.txt
├── get
│   ├── app.py
│   └── requirements.txt
├── list
│   ├── app.py
│   └── requirements.txt
├── swagger.yaml
├── template.yaml
└── update
    ├── app.py
    └── requirements.txt

This looks like a good comproise. You could have swagger.yaml self-container for a resource , same for template.yaml. But how could we need merge those yamls together?

we could use some yaml merging tools like yamlinc and swagger tools also. AWS SAM support include swagger.yaml file but only that file is somehere in s3 which is url - so it is still not 100% integrated in dev process.

All those means we need change your build process little bit and we might need write some custom Makefile etc to achieve that, but it should be possible if we could achieve better development, easy maintance and so on.

Notes

Traffic control

Pricing is big consideration. But we need think before that, in a systematic way. What might be the bottleneck of the system? We need control the flow and spikes so it wouldn’t take down service or spinning too many instances with no-availbility at all. Here the IoT, user will frequently upload there gps coordinates, how we prepare for that? we could Kinesis or simple SQS to do the aborbe and fine control the rate.

Also we need understand about the Lambda’s burst model, cold-start and warm start and do some benchmark to get reason memory/cpu allocated for best performances and good money-savings.

In traditional architecture, the database (relational one) or disk-related file writing/reading are kinda bottleneck. We might try to use s3 or nosql database for most time. But if we need use that, e.g. postgresql/mysql(RDS), we need take extra care for not allowing database become the bottleneck of system.

For example, a VPC does sound good idea, but in somecase, if not used properly it will exhaust the ip pool and cause lambda keep piling up (if not configured and twisted properly). Also VPC could slow down the cold start time significantly.

Read through documentation

There are quite lots of concept and techs in AWS serverless if you, like me, never learned that before. AWS has lots of greate resources online. The aws re-invent and aws online talks are pretty good, covering tons of topics, which is really good opportunity to learn. I found interesting about take new skills or techs is that we all gonna go through following stages:

step1: follow some tutorials
step2: get hands dirty , play with demos
step3: okay, I got it basically, then can’t wait start real coding
step4: xxx

There is an impulse/urge to jump to detail implementation - instant gratification for self-proving ? But often later, we got caught in big hole and sometimes got driven crazy as felt no way to get out of it, esp, when time is not on your side. Maybe you need deliver some features before deadline?

How many times have I made those similar mistakes, not only in coding but other area of life too?

After step1 and step2, you’re good, as you know basically how it works. And jump to nuts and bolts is fine. What’s wrong? you need step back and look from a big view when needed. Always go back to basic and origin, take some dumb time to read through documentations always saves me big time later. No one likes to read docs, (like APIGateway, Lambda, IAM, CloudFormation, SAM, CodeBuild, CodePipeline, CodeDeploy), it is not fun at all. But it is a must, you could try with comparing to tech/skill/language that you already familiar with, yeah, diverting your attention and make it little bit more fun.

A good programmer should be good to always jump back and forth between abstraction and implementation.

Anyway I think AWS Serverless is pretty fun and interesting tech to learn and use. Hope they could improve their ecosystem around it to provide better development experience in the future.

Ant Design Pro(v4)中Tabs使用

2019-11-03T13:54:32+00:00

最近因为需要在项目中引入了Ant Design Pro(v4)，其中有一个页面需要在顶部添加Tabs.类似于https://preview.pro.ant.design/profile/advanced

Ant design里关于Tab组件的使用Tabs标签页

何时使用# 提供平级的区域将大块内容进行收纳和展现，保持界面整洁。

Ant Design 依次提供了三级选项卡，分别用于不同的场景。

卡片式的页签，提供可关闭的样式，常用于容器顶部。

标准线条式页签，用于容器内部的主功能切换，这是最常用的 Tabs。

RadioButton 可作为更次级的页签来使用。

还是蛮清楚简单的。

所以关于使用方式常见的一般分三种情形：一种是顶部标题不变,底下tab切换变化，URL不变；一种是底下tab的变化对应顶部的变化；最后是顶部不变，tabs切换，但是URL需要变化。

我们讨论最后一种，因为我们可以想象Tabs的内容里是常见的表格，然后可以查看详情跳到另外一个页面，这个时候从那个页面返回就需要回退到之前所在的tab；同时用户有可能直接收藏跳到该tab的需要。

分析

我们想达到这个效果:

http://localhost:8000/project/user/230/wishlist:

http://localhost:8000/project/user/230/detail:

那代码这边如何处理了？

Ant Design Pro在v4里将路由改成了可配置的方式，将这些scaffolding的事情都交给了umi, 所以基本上你只需要在config下面配置路由就好.(路由和菜单)

{
    name: '详情',
    path: '/project/user/:id/detail',
    component: './project/user/detail',
},
{
    name: '心愿单',
    path: '/project/user/:id/wishlist',
    component: './project/user/detail',
},

这里我们配置了相同的component,因为我们希望他们都经过这个HOF来先处理，比如共同的PageHeaderWrapper，还有就是如何render当前的tabs, 最后就是通过路由匹配来懒加载其真正内容的组件。

所以我们就得先看看这个入口的detail组件如何实现： 1.Convention over Configuration如何从路由列表中解析出当前的所有tabs; 2. 如何或者以何种规则加载其内容组件.

这是入口文件，可以先看左边的目录。Ant Design Pro组织目录方面是根据模块来划分的，理论上每个模块无非都是UI渲染还有数据获取管理这两个核心功能，所以它一般都有index.jsx, service, model等同一的结构来划分不同的关注点。

首先我们看看如何解析路由, 先获取当前路由的match.假使你访问http://localhost:8000/project/user/230/wishlist:

isExact: true
params: {id: "230"}
path: "/project/user/:id/wishlist"
url: "/project/user/230/wishlist"

这个时候我们通过match.path去遍历路由配置来获取匹配/project/user/:id的所有路由。

function findRoute(basePath, routerData, result){
	if(_.isEmpty(routerData)){
		return null;
	}
	routerData.forEach(item => {
		if(!item.path){
			return
		}

		const basePathFromCurrent = removeLastPart(item.path);
		if(!(basePath.startsWith(item.path) || basePath.startsWith(basePathFromCurrent))){
			return
		}
		//if path without last part match with current path, then it is the one
		if(basePath === basePathFromCurrent){
			let lastSegmentPath = lastSegment(item.path);
			const realComponentName = capitalize(lastSegmentPath);
			result.push({ ...item, realComponentName, tabKey: lastSegmentPath })
			return result;
		}
		findRoute(basePath, item.routes, result);
	});
	}

/*
	name: '详情',	
	path: '/project/user/:id/detail',
	component: './project/user/detail',
	realComponentName: 'Detail',
	tabKey: detail
* */
export function getRoutes(path, routerData=menuConfig) {
	const basePath = removeLastPart(path);
	let result = []
	findRoute(basePath, routerData, result);    
	return result
}

原来v1是有这个getRoutes的但是后面因为挪到了umi里，这个就没有了。这里是没办法直接拿过来直接用的，所以我自己快速写了一个简单的，当然代码还是可以优化。

所以我们得到了一个变形过了的数据回复，每一个元素都额外有了两个属性： realComponentName和tabKey。这个realComponentName就是url最后一部分wishlist/detail首字母大写，然后tabKey就是url last segment, tab的名字就是用name.

然后我们在tab切换的时候，只要拿到当前url的基础部分，加上当前的tabKey，直接push就可以了。

onTabChange = key => {
    const {dispatch, match} = this.props;
    const base = removeLastPart(match.url)
    console.log(`${base}/${key}`)
    dispatch(routerRedux.push(`${base}/${key}`));
};

关于内容加载部分就比较简单了，我们用到了React的新属性功能： lazy&Suspense. Code-Splitting

 <Switch>
    <Suspense fallback={<div>Loading...</div>}>
        {routes.map(item => (
            <Route key={item.path} path={item.path} 
                component={React.lazy(() => import(`./${item.realComponentName}`))} 
                exact={item.exact}/>
        ))}
    </Suspense>
</Switch>    

到这里，我们只需要在当前目录下添加capitalized的文件名的组件即可，它可以自动被识别加载。

这样我们就通过菜单，自动识别创建tabs，直接添加组件就可以了，无需手动参与配置。

引申

这个思路在于发问现实世界里重复性的或者说需要耗费脑力的工作是否可以通过更好的方式和实践来减少。我们要认识我们大脑习惯性的自动巡航，而且从大部分情况来说是非常有益的，让我们能更好的将精力花在值得的更重要的事情上。所以我们开发必须以服务自动巡航这个客观规律为基准，来提高我们日常的开发效率。

减少非核心非业务逻辑的大脑消耗，这样就能更好集中精力在梳理和实现业务逻辑。

回想一下我们后端的目录设计，普遍的基于Node的，比如express，都会分成model, routes, controller等等几个层次，每个层次都有具体的业务模型。比如我现在要修改User的某个字段或者逻辑，我不得不经常切换与controller/routes/model等等而且他们名字通常都一样，即使借助于IDE的一些快捷键，有时候还是需要切换上下文，非常容易卡壳。专注业务逻辑实现时候，不得不因为目录或者项目组织的原因，挂起那进程，开启新的线程，即使很小的开销，也会影响节奏。

节奏感很重要，除了如何更好组件项目让业务实现能顺畅，一些不同维度的具体实践也有很帮助，比如熟记快捷键(Jetbrain有统一的一套的快捷键Refactor&Navi，只要记住一遍，基本能覆盖所有语言），比如黏贴版工具（Jumpcut，快捷键切换）等等。

所以我们后端的目录构造，跟前端是相对应的。这样一来更容易找到更某个模块相关联的代码和逻辑，脑子不需要多余的开销就能顺畅找到它想要的，丝滑般。。。。smoooooooth, bro

在model级别，我们规定文件名需要以model.js结尾，路由则已route.js结尾。如果是跟前端相关的API比如给小程序或者h5使用的，在frontend.route.js里面写；跟后端管理相关的API，则写在dashboard.route.js里。这里我简化了controller跟业务逻辑的严格区分，实际上你可以有一个service级别来严格做业务逻辑。

那么如何在项目启动时候识别加载这些我们定义好的convention了? 其实很简单，比如就是说Sequelize的model把。

const sequelize = new Sequelize(config.database, config.username, config.password, config);
let routerDir = `${__dirname}/../../routes/`;
glob
    .sync('**/*model.js', {cwd: routerDir})
    .forEach(file => {
        //console.log("filedb: ", file)
        const model = sequelize["import"](path.join(routerDir, file));
        db[model.name] = model;
    });
Object.keys(db).forEach(modelName => {
    if (db[modelName].associate) {
        db[modelName].associate(db);
    }
});

glob一下而已。

关于路由这块，以前按照frontend/dashboard来分，我们可以很简单做jwt的事情，那现在如何办了？

let rootRouter = Router({mergeParams: true});
rootRouter.use("/dashboard/", jwt({
    secret: config.jwtDashboardSecret,
    credentialsRequired: true,
    requestProperty: 'user'  //use req.user to get staff
}).unless({
    path: [
        {url: /^\/api\/dashboard\/auth\/.*/},
    ]
}));

rootRouter.use("/frontend/", jwt({
    secret: config.jwtFrontendSecret,
    credentialsRequired: true,
    requestProperty: 'user'  //use req.user to get user current
}).unless({
    path: [
        {url: /^\/api\/frontend\/auth\/.*/},
    ]
}));

也是非常简单的。

这样一来，我们保证了前后端近乎相同的目录结构，按照责任划分组织，更好的服务于了现实世界的需要，同时也让程序员更好的专注在核心开发上。

辩证唯物主义和行为心理学

2019-09-07T13:54:32+00:00

最近看了一本很久之前买的二手书 1983 年的《论毛泽东哲学思想》，之前一直没看也是因为读书时代的死记硬背那种感觉“空话虚无缥缈”让人心里有抵触，一方面可能因为其晚年建国之后的失败，难以静下心来。总而言之，被自己的潜意识或者第一系统形成的认知模式负面影响，没有调查就没有发言权，在这个时候就很难了。后面我意识到了自己的这个抵触心理，违反了开放心态原则，为什么我有这个抵触心理？这个是反映了真实情况么？如果我抛开偏见和潜意识的影响，联系他当时的实际和自己的实际，能从其中学到什么么？

看完之后，惊为天人，醍醐灌顶，茅塞顿开，看来毛选得买一本,然后就是马克思了。

物质-精神-物质，实践-认识-实践

毛泽东可以说在那个动乱复杂的年代，在资讯相当不发达的年代，有这样一个开放的心态，学习吸收有限的资料来源（翻译的马克思资本论），在恶劣的形势下实践中不断总结主动独立思考，并形成一个套适合中国特色国情的方法论，这一点真的令人佩服。特别是他关于认知方面（主客观），还有将其运用在军事战略方面，真是叹为观止。在无数次形势危急的时刻，别人普遍悲观丧气时，他却有一种无与伦比的坚定，有比黑暗中的火苗，引领着其他人，令人深思。我们必须学习他如何思考和如何看世界的方法来指导我们日常生活。

湖湘学派从王夫之到曾国藩，经世致用，内圣外王，毛泽东青年时候看起来就对哲学很早就有接触认识的。

于近人独服曾文正”，其理由之一是曾氏能探得“大本大源”

大本源也就是道，他对《道德经》老庄墨等等研究运用的很深刻。对大本源的思考，也让他能从更高的角度看问题，同时善于从实践中总结抽象普遍规律和方法，反过来更好作用于实践。

认知

人们在社会实践中从事各项斗争，有了丰富的经验，有成功的，有失败的。无数客观外界的现象通过人的眼、耳、鼻、舌、身这五个官能反映到自己的头脑中来，开始是感性认识。这种感性认识的材料积累多了，就会产生一个飞跃，变成了理性认识，这就是思想。这是一个认识过程。这是整个认识过程的第一个阶段，即由客观物质到主观精神的阶段，由存在到思想的阶段。这时候的精神、思想(包括理论、政策、计划、办法)是否正确地反映了客观外界的规律，还是没有证明的，还不能确定是否正确，然后又有认识过程的第二个阶段，即由精神到物质的阶段，由思想到存在的阶段，这就是把第一个阶段得到的认识放到社会实践中去，看这些理论、政策、计划、办法等等是否能得到预期的成功。

人们的认识经过实践的考验，又会产生一个飞跃。这次飞跃，比起前一次飞跃来，意义更加伟大。因为只有这一次飞跃，才能证明认识的第一次飞跃，即从客观外界的反映过程中得到的思想、理论、政策、计划、办法等等，究竟是正确的还是错误的，此外再无别的检验真理的办法。而无产阶级认识世界的目的，只是为了改造世界，此外再无别的目的。一个正确的认识，往往需要经过由物质到精神，由精神到物质，即由实践到认识，由认识到实践这样多次的反复，才能够完成。这就是马克思主义的认识论，就是辩证唯物论的认识论。

自觉的能动性既包括认识，又包括实践。自觉的能动性既表现在从感性认识到理性认识的飞跃，也表现在理性认识能动地指导实践。

首先从感性到理性，主观上提高。

读书是我终身的爱好

读书学习是人生的第一需求。主观能动可以通过学习，学习自己的经验，当然也可以学习历史中西。读书就是一个很好的来源。

美国作家哈里森·索尔兹伯里在《长征：闻所未闻的故事》一书当中评价毛泽东的卓越才华时这样写道：“他毕业时 25 岁，谙熟孔子、孟子和中国的伟大文学，他懂得佛教和西方哲学，他学习了美国和欧洲的政治、地理，他吸收了中国古典的军事智慧和孙中山先生的改良思想。他能言善辩，是位诗人、爱国者，是位年轻但迅速成熟的哲学家。”

我比较惊叹他对于新知识的接受程度（可能跟魏源等等都有关系），一点没有守旧的老思想，接受新的西方思想；同时也善于从中国古代经典中吸取营养；开放的心态，能我所用者都可以拿来，务实主义，不偏颇，用他的话说：“批判性的读书，批判性的学习，批判性的吸收”。在长征枪林弹雨或过雪山等等无论形势多恶劣，顺境逆境，他也是想尽办法读书，这点终身学习的思想值得我学习。

毛泽东曾这样描述他在湖南一师图书馆读书的感受：“我就贪婪地读，拼命地读，就像牛闯进了人家的菜园，尝到了菜的味道，就拼命地吃个不停一样。”

这点就讲到了现在当代生活知识触手可得，有了手机，感觉随时随地都能获得。好比纽约本地人不去看跨年蛋，上海本地人不去迎新跨年，当你有了这个东西在了后备箱，心理上反而不会去注意它的存在。后面会讲到信息爆炸年代对于短期感官刺激的强烈推崇，我们需要特别注意。

在《西行漫记》中，斯诺写到： “毛泽东是个认真研究哲学的人。我有一阵子每天晚上都去见他，向他采访共产主义的党史。一次有个客人带了几本哲学书给他，于是毛泽东就要求我改期再谈。他花了三四夜的功夫专心读了这几本书。在这期间，他似乎是什么都不管了。他读书的范围不仅限于马克思主义的哲学家，而且也读过一些西方的哲学家——斯宾诺莎、康德、黑格尔、卢梭等人的著作。”斯诺惊叹地说：“不可否认，你觉得他的身上有一种天命的力量。这并不是什么昙花一现的东西，而是一种实实在在的根本活力。

韩愈说：‘人不通古今，马牛而襟裾‘。读历史太重要了，历史不会重演但总是惊人的相似。

笔记总结

主观上读书自然有用，但是读书需要结合人物特定的情景时代来结合看，才能有设身处地的感悟。要勤于做笔记，勤于总结，联系实际，才能转化为自己的东西，这个主观才是进化的主观。结合《矛盾论》里的方法论，从整体局部，前后左右不同立体的角度，主要次要等等角度来思考总结，更好的批判性吸收。

有些时候有的新手程序员不爱做笔记，完全靠脑子记忆，但很多时候了记忆是会消散很快的。

“Ideas can come from anywhere and at any time. The problem with making mental notes is that the ink fades very rapidly.” ― Rolf Smith

记笔记不一定需要李敖那样撕书了，毕竟也都电子化了。但是可以利用简单的，比如便签啊索引卡（电子和实体都可以，有道啊印象啊或者自带的日记等等）。这样做的好处在于快速记下不清楚的点或者有疑问的点（用最简单的方式），后面得空再来深入了解。

之前有记得看到一个作者 Ryan Holiday 写过一篇文章The Incredible Creative Power of the Index Card: An old-school tool that rivals its digital competitors，讲到他如何在平时收集自己的零星的想法，就用到了 Index Card(索引卡)。然后在归门别类，方便后续系统的翻看或者回顾整理。

里根的幽默和段子频出在美国总统上应该排的上号了。除了总统再次竞选时，被主持人质疑年纪过大时（肯尼迪应该是最年轻的了 40 多岁，而里根当时 70 多，跟现在特朗普和拜登都没得比，当时算是年纪很大了，特别是第一次时 69 岁，已经是历史上年纪最大的总统），可能无法像肯尼迪在古巴危机中连续不停工作，他机制的回答自己不会利用自己年纪大的经验优势来 exploit 占对手年轻没经验的便宜，引得了观众甚至对手的大笑，巧妙的化解了年纪问题。Reagan-Mondale debate: the age issue

里根讲了不少吐槽关于苏联的段子, 其中一个：

一個俄國男子要買一輛汽車，車行說，他要等10年才能交貨。

男子：「是早上還是下午？」

車行：「你要等10年以後呢，是早上還是下午有什麼區別？」

男子：「啊，因為管工要早上來。」（包袱：管工也要等10年）

他自己平时一直收集段子，其中还有一个花招是在闲暇时在索引卡上写一些 One-Liner(很短的笑话，效果出奇的好), 在 Reagan Libray 的一篇纪录片Ronald Reagan’s one-liners里提到他的这个习惯：

记笔记还是挺好的习惯的。

平时不用心记录笔记，比如可以是印象笔记或者有道笔记，把临时没有时间能深挖的点或灵感记下来，后面你如何能够将零散的知识点进行总结深挖并编织成一张网，一个知识体系了。写个博客就是这么简单的事情，平时记下关键点在笔记上，等到空下来仔细研究，然后画个脑图梳理关系逻辑，这博客不就出来了么？难么？不难。只是不够用心罢了。

当你有无数个这样小网，加上时间先生的酝酿和大脑潜意识的魔力，你就能将这个无数的小网融会贯通编织成一张大网，一张能够更好应对各种形势的网。

没有笔记，没有总结，何来主观能动的有效提高，更加不可能能有效的认识客观并且改造客观。悟，必须是有输入的，才能悟。

等你养成这个习惯，面对新的事物，你也不会毫无头绪，你知道如何归纳整理，不就是一个很好的应对变化的系统了么？

“一切带原则性的军事规律，或军事理论，都是前人或今人做的关于过去战争经验的总结。这些过去的战争所留给我们的血的教训，应该着重地学习它。这是一件事。然而还有一件事，即是从自己经验中考证这些结论，吸收那些用得着的东西，拒绝那些用不着的东西，增加那些自己所特有的东西。这后一件事是十分重要的，不这样做，我们就不能指导战争。

批判性吸收，总结写文字写博客就是一个迫使自己主要思考，引入第二系统的过程（思考快与慢），打破思维惯性，联系自己的实际，系统化吸收消化的过程，非常有效的方式。

实践

毛泽东说：“向着最坏的一种可能性做准备是完全必要的，但这不是抛弃好的可能性，而正是为着争取好的可能性并使之变为现实性的一个条件。”毛泽东在战争指导中，往往立足于战争的最坏的可能性，并设法创造条件，争取最好的可能性。

比如济南战役:

你们的根本出发点应放在最困难的情况上，依情况发展，如援敌进得慢，而攻城进展顺利，又有内应条件，则可考虑增强攻城兵力，先攻城，后打援；如援敌进得快，则应以全力先打援，后攻城。

联系实际，比如开发之中，除了自己学习变好，在项目紧张之中，还需要给新手的程序员制定好他的学习计划，并且花时间追踪，帮助他成长。看似是花费了实现更多项目的可能，但是实际上这是防守中的进攻，进攻中的防守，有了这个我们的防御和进攻会更犀利，从大局上讲，是非常划得来的。真的进攻不是一定在于一定要一直往前冲，退一步可能是为了走的更快更长远。

Ps: 这里可以贴上我给新手程序员其中之一回顾并制定的如何培养好的习惯，包括学习思考总结，最重要是构建好的认知体系，应对不断变化的可能。以己度人，必须思考如何让他变得更优秀，应对除了程序代码之外的能力，最重要是学会主动思考和好的习惯来应对生活，对他的人生也会有帮助，其中也会给我带来很多启发，一举两得。

实践、认识、再实践、再认识，这样形式，循环往复以至无穷，而实践和认识之每一循环的内容，都比较地进到了高一级的程度。

这不是正面效应的叠加，反身性里的正面加强， Ray Dalio 的那个进步周期循环。实践是主观作用于客观的，第二个阶段。

读书是学习，使用也是学习，而且是更重要的学习。从战争学习战争——这是我们的主要方法。没有进学校机会的人，仍然可以学习战争，就是从战争中学习。革命战争是民众的事，常常不是先学好了再干，而是干起来再学习，干就是学习。

大学和人生比，只是一个小小的阶段。读书学习实践，人生生活才是最大的试验场，不可抱残守缺，更不可固步自封。独立思考，比如有源才有可能，没有大量的输入并且有有效的转换方式，加上长期的迭代，是不可能形成独立思考的。而漫长却又纷繁复杂变幻的人生之旅，就是最好的修炼场，给了一个绝佳的试验田。

在《中国革命战争的战略问题》一文中，他说：“指挥员的正确的部署来源于正确的决心，正确的决心来源于正确的判断，正确的判断来源于周到的和必要的侦察，和对于各种侦察材料的联贯起来的思索。指挥员使用一切可能的和必要的侦察手段，将侦察得来的敌方情况的各种材料加以去粗取精、去伪存真、由此及彼、由表及里的思索，然后将自己方面的情况加上去，研究双方的对比和相互的关系，因而构成判断，定下决心，作出计划，——这是军事家在作出每一个战略、战役或战斗的计划之前的一个整个的认识情况的过程。

毛主席指出：“认识情况的过程，不但存在于军事计划建立之前，而且存在于军事计划建立之后。当执行某一计划时，从开始执行起，到战局终点止，这是又一个认识情况的过程，即实行过程。此时，第一个过程中的东西是否符合于实况，需要重新加以检查。如果计划和情况不相符，或者不完全符合，就必须依照新的认识，构成新的判断，定下新的决心，把已定计划加以改变。使之适应于新的情况。部分地改变的事差不多每一作战都是有的，全部地改变的事也是间或有的。鲁莽家不知改变，或不愿改变，只是一味盲干，结果非碰壁不可。”这就是典型的让主观服从客观的辨证唯物主义的哲学观。

矛盾一直在变化之中，但不是没有规律的。要认识规律，必须认识矛盾。需要实地调查。

没有调查就没有发言权

这里我们思考，为什么比如在别人讲话的时候容易打断别人？

为什么我不是在倾听完之后思考，然后回答。别人说的话，我们有没有引入第二系统的理性思维去思考判断他讲的东西是否是客观情况，而不是任由第一系统感性，介于感性受到的影响因素太多了，波动特别大（比如情绪），会严重影响我们的判断，脱离事物的本质。更多时候争论变成了由实际主题偏离到无关之处，上身到人生攻击，最后甚至打高。

没有调查就没有发言权，这点实属牛逼。难的地方在于受制于每个人的认知不一样，每个人的经历不一样形成的认知体系包括自尊等等，还有行为心理的不同，决策是非常容易受干扰的，脱离实际，任由经验主义或者情绪绑架我们，看起来是普遍存在的。好比我读这本书之前，我的经历让我一直心理抵触，又或者我刚刚吵了一架还未平复，根本提不起兴趣。刚开始我心里觉得是不值一读，但是实际是不是了？我有没有实际调查了？没有。为什么没有？因为决策系统和行为系统在主观上出现了偏差。

客观上来，他建国之前的成功是因为他坚持了他的哲学思想，建国后的失败是因为他没有坚持他的哲学思想。由于自己的骄傲自大，脱离了客观实际，种种原因没有实地调研，造成了主观上的唯心主义和行为上学，失败了。但是这不正有效的说明毛泽东哲学思想的伟大正确之处？

辩证唯物法很重要很有用，从《实践论》和《矛盾论》提供了一个非常有价值的方法论系统，特别是研究他的时代情景等等综合考量，他如何思维和思考让人备受启发。

在当今时代，我们如何将毛泽东的哲学思想更好融入现代生活，提高主观的认识，更好的服务客观，提高二者对立统一波浪式前进的效率了？

实事求是。我们需要结合当今时代特点，来分析，特别是行为心理学和决策学。

行为心理学和决策过程

70 年代以来的心理学的研究和科技的进步，人类对大脑有了更深的认识。对于主观个体来说，如何更好跟客观保持同步，是非常值得思考的。主观个体需要更好的了解自己，了解人性，方可更好的认识客观，反过头来更好从客观实践中改造深刻自己的认识。

主观和客观必须保持平衡，这个客观因为内在的矛盾又是动态的，所以必须不断的检视自己的主观认识，确保自己不脱离客观情况，不固步自封，更不可骄傲自大，当然也不能妄自菲薄。真理永远在变化之中，只有守住中庸的底线，认识自己，（这就包括如何认知自我，情绪，潜意识等等如何影响我们的理性判断）

1970 年代早期 Maxwell Maltz 的《Psycho Cybernetics》，虽然名字是奇怪了点，但是内容确实不错。然后就是 Eckhart Tolle 的《The Power of Now》当下的力量。

卡尼曼的《思考,快与慢》是一本很好的书, 关于 behavioral psychology and decision-making，特别是里面讲的两种思考系统：System 1 (Thinking Fast), and System 2 (Thinking Slow). 第一系统是本能反应 gut reaction 基于经验潜意识，基于第一印象直接跳到结论（比如先入为主），快但不准确；第二系统是理性的分析思考过程，问题解决导向，慢费力但是相对准确。大部分时间是系统一的思维，对于熟悉的事物，不需要主动引入思考，快速省力，但是如果碰到了不熟悉的事物，可能第二系统就会介入，分析解决问题。里面还列举了很多思维决策的 cognitive bias.比如过度乐观，就会任由第一系统放大，认为自己无所不能，而不去沉下心来检视客观条件的变化，脱离了客观，必然会失败。比如在情绪低落时，大脑会自动联想起很久之前的负面体验，来强化你的沮丧感，放大你逃生的本能，可能实际情况没那么糟糕，但是你已经是如坐针毡。

人们通常形成第一系统的印象，比如我觉得 xxx,有了这个看法，然后去搜索片面的对自己第一系统产生的想法有利的客观证据或条件，而有意忽视或者不愿意（可能找借口）来寻找跟这个看法相反的客观事实。为啥了？因为我们祖先这样省力能更好的的生存在危险遍布的非洲的大草原上，大部分情况下。

我们的本能在这个年代受到的很大的冲击，本能是需要检视的，不能放任其控制我们的思维过程。银行 or 国家 Too big too fail,不存在的，本能 Too long to fail, 更加不存在，我们需要实事求是，灵活变动。

这就为什么李小龙说：

Empty your mind, be formless. Shapeless, like water. If you put water into a cup, it becomes the cup. You put water into a bottle and it becomes the bottle. You put it in a teapot it becomes the teapot. Now, water can flow or it can crash. Be water my friend.

这句话是普遍真理，但是难，对抗本能。我们需要认识到本能是可以被改造的，可以像健身一样可以被训练的。

首先要识别，识别本能的迹象，在本能反应之前引入一丝丝理性的空间。

情绪和身体感官的感受使我们跟大自然跟宇宙的连接的桥梁和通道。我们的本能会通过感受传递出来，比如某个人说的一段话，可能无心，可能是就事论事，但是触发了你童年时期某一个心里阴影，在你不自觉的情况下，你被激怒，心跳加快，脸色发红，全身绷紧，随时准备反击，好比老虎一样，这个别人说的是否客观，不重要了，你要证明保护捍卫自己的生存。

自尊是本能是很重要的一个假象，自尊大部分下随着年纪渐长，很多变得像是一个紧箍咒。像毛泽东说的是一个纸老虎，你越加在意，它被吹的就越大，你就越恐怖；反过头来，你不去在意它了，它不能再是你的身份象征了，它这个纸老虎就不攻自破。

除了更好认识自我自尊的关系，通过一些具体实践我们可以加强对身体感官感受的认识和联系，更好知道自己处在什么阶段，让自己的反应能得到一点点理性化。

多回归自然，放下手机，加强对呼吸的练习，比如深呼吸，多观察万事万物本真，是一些很好的办法；当然冥想瑜伽太极拳都是，万法归一。

这个时代，资讯爆炸，手机的影响太大，特别当你是程序员，每天基本都是电脑或者手机。你可以看到生活那么广告，那么超大的字体，要么不断重复的话语，不断的冲击刺激你的大脑，潜移默化的影响你的第一系统。

第一系统的要务是生存，而且是短期的刺激和满足。这个形式可以说是毛泽东应该没经历过的，但是他当时也没有经过如此动乱的年代吧，所以我们可以从他身上和他历程中学习到什么。所以我觉得我应该识别短期刺激的本质，第一系统的认知偏见，和本能反应这套机制，然后因势利导，把不利变成利，给它转换过来。

任凭风浪起,稳坐钓鱼台

短期的刺激自然引起大量追捧，那么给了长期更好的价值，所以我们要识别短期的疯狂，藐视它，战略上重视她，然后立足短期把握长期，从更高的角度来思考，更长远的目光来审视，这样理论性如果短期越被疯狂消费，那么长期的积攒的效应就会越大。

在诱惑面前每个人都有原始本能的体现，没人能控制的住，但是我们可以认识识别，一旦我的第一直觉是这样那么大部分人肯能都是这个反应我反而要利用这信号，并且倾听懂身体感官感受的信号，提醒自己转而思考对立面，并问自己我有调查这个问题或者现象么？如果我能有一个短暂的缓冲空间，哪怕片段，也能引入我们的理性思维，从而做出最符合客观情况的判断。

相比几百万年的时长，近一百多年的工业化和信息化数字化的时间段实在是太短暂了，大脑的构造和原始本能根本无法跟上这么快的变化。原来必须实际付出劳动，通过手脚或者多人合作加上实体工具的运用，才能地艰苦的生存，信息的通信沟通和渠道都是非常困难，意味着“等”和单调可能是大部分时间的状态；现在变成了虚拟的了，温饱也基本不用太多担心，信息触手可得，”快“和”复杂“可能是大部分时间下的状态了。原始本能貌似失去了优势，但是事情必须辩证的来看，这些虚拟的短快的东西未必有营养，未必能沉淀下来，没有经过实体原始的身体感官，没有了对时间的耐心和敬畏，我们未必能消化的更好，相反可能带来的更多的困扰。所以 journal 记笔记（私人笔记），手写的那种，像是学生时代暑假寒假那种记日记的方式，反而是充分利用了原始本能的优势，通过运用身体，工具和大脑合二为一，脑手笔更像是一体的，就像原始人本能求生存那种，让思维想法更好的得到组织流动整理，好比重新翻整耕种歇息过的耕地一样，更好的提供营养。返璞归真，才能更好的让我们看清短期的迷雾，才能更好让我们看清冰山下面的部分。我觉得手写的这种比较适合结合定期的一个反思反省总结（Reflect)，效果更好了。

1933 年富兰克林-罗斯福第一次就任美国总统，时值美国大萧条刚刚发生没几年，失业率和经济形势相当恶劣：

我肯定，同胞们都期待我在就任总统时，会像我国目前形势所要求的那样，坦率而果断地向他们讲话。现在正是但白、勇敢地说出实话，说出全部实话的最好时刻，我们不必畏首畏尾，不着老实实面对我国今天的情况，这个伟大的国家会一如既住地坚持下去，它会复兴和繁荣起来。因此，让我首先表明我的坚定信念：我们唯一下得不害怕的就是害怕本身——一种莫明其妙的、丧失理智的、毫无根据的恐惧，它会把转退为进所需的种种努力化为泡影。凡在我国生活阴云密布的时刻，坦率而有活力的领导都得到过人民的理解和支持，从而为胜利准备了必不可少的条件。我相信，在目前危急时刻，大家会再次给予同样的支持。我和你们都要以这种槽神，来面对我们共同的困难。感谢上帝，这些困难只是物质方面的。价值难以想象地贬缩了；课税增加了，我们的支付能力下降了；各级政府面临着严重的收入短缺；交换手段在贸易过程中遭到了冻结；工业企业枯萎的落叶到处可见；农场主的产品找不到销路；千家万户多年的积蓄付之东流。

其中：

So, first of all, let me assert my firm belief that the only thing we have to fear is fear itself–nameless, unreasoning, unjustified terror which paralyzes needed efforts to convert retreat into advance.

情绪本身并不可怕，问题是没有理性介入，情绪变成了催化剂，加速向下或向上的作用(down/upward-spiral)。所以我们首先要识别，情绪令人不安，不安则由本能逃生而起，大脑发出强烈的信号加速身体的反应，我们会不自然的本能想快速逃离或者平息来获得情绪或者生理心理上的短暂的安全或者稳定感，但是殊不知，大部分情况下，这个“短期的稳定安全感“会以长期为代价，往往陷入更大的问题或者麻烦中，带来更大的损失，造成-往往你想不到的-更大的心里生理的痛感。于是你悲愤，无奈，丧失了自己，也打破最后的壁垒，不得不重新坦诚的接受现实，也许能获得认知上的革新，但是有时候代价未免有点大。

识别->停顿->反应。我们在情绪之中时要避免做决定，识别，但是不要立马反应，因为情绪或者短暂的身体感官感受就好像空中的云朵一样，只要你不去放大它，刺激它，它自然而然会自动消散，它无法持续很长的时间（太费精力了，身体脑力都支持不了，我能时时刻刻每分每秒都在发怒或者狂喜了？不可能的,生存本能也注定了这个是不可能的），只要意识到这点，转移注意力或者调控你的身体过感官（比如深呼吸，比如走出来散一步，远离刺激源），This too shall pass, 心静下来，好比道家和儒家说的”静生定,定生慧,慧至从容“，有了这个静就能引入关键的理性分析（我有调查了客观情况，我是否掌握了确切的当前事物的情况），从而客观分析，为此才能跳脱出恶性循环。另外这个情绪反应刚好给了我们一个非常重要的反向指标，来观察大众如何思考问题，并如何影响事物的走向，这点对我们后面的理性决策提供了非常重要的参考，能更好的帮助我们还原客观的本质。

ps:美国历任总统就职典礼的演讲稿结合人物生平和时代背景，关联起来还是非常有借鉴学习作用的，非常值得一读。

《大学》里说：

孟献子曰：“畜马乘，不察于鸡豚；伐冰之家，不畜牛羊；百乘之家，不畜聚敛之臣。与其有聚敛之臣，宁有盗臣。”此谓国不以利为利，以义为利也。

作为程序员，我每天下午会抽出十分钟，放下手机，走到楼下。看看天空和周边，慢慢闭眼深呼吸吐纳 10 次，然后后退走路，走路之时观察身体的感受，脚啊腿啊腰背等等，感知风拂过或者周边生活各种声音，挺有意思，这样既有冥想之效又可有锻炼非惯性思维，一举两得，对于大脑而言，好比一个按摩。

我很推荐程序员 Andy Hunt 大佬的《程序员的思维修炼: 开发认知潜能的九堂课》，总结的很好，每个程序员都值得一读，术与道结合才能应对变化。如果你不能认知自我，了解客观规律，主动学习思考总结，那么你就无法把握事物的核心矛盾，抓不住主次，就容易陷入局部的死循环，更何谈应对风险（你自己可能就比风险还可怕），所以程序员一定要学会主动独立的思考。

一旦每天都作，形成一个习惯 routine,那么长期来看必定是会有裨益。记住事物是波浪形的前进，进攻中的防御是为了更好的进攻。

心地清净方为道，退步原来是向前。

我还是要讲到习惯->行为->身份/信念，这三个是互相作用，里外统一的。我们必须建立自知的习惯，通过行为来强化，最后升级到身份或者信念，最犀利的是反过头它会更好指导行为习惯。我觉得要是想通了，那么习惯无法坚持下去的接口都不是借口，困难只是一个更好机会来锻炼自己的心态，（如果每一次都一样，那就不是事物发展的本质了），所以当你疲惫或者懈怠时，不妨给这个疲惫和懈怠 20%的额外的动力，事情自然就能坚持下来。

毛泽东的军事，马克思的经济结合两者的哲学指导思想，加上古典的道家佛家（禅），西方的古希腊雅典苏格拉底柏拉图，古罗马的斯托葛派，把这些思想整合，保持开放的心态学习，结合历史进程，批判地吸收总结实践，并融入个人的日常生活的具体活动之中（可以是比如马克思主义的搬砖写代码或者啥的），这里就是结合个人的个情了，不断进行认识->实践->认识的迭代，总结反思，想必很有意思的。

开放学习，融会贯通

王国维的《人间词话七则》中有一段：

大家之作，其言情也必沁人心脾，其写景也必豁人耳目。其辞脱口而出，无矫揉妆束之态。以其所见者真，所知者深也。诗词皆然。持此以衡古今之作者，可无大误也。诗人对宇宙人生，须入乎其内，又须出乎其外。入乎其内，故能写之。出乎其外，故能观之。入乎其内，故有生气。出乎其外，故有高致。

约翰•梅纳德•凯恩斯:

学习经济学，似乎不需要什么高度的持有的天资。从智力上来看，跟哲学或纯科学的一些学科比起来，不是很容易吗？这门学科看起来容易，但是能学得出人头地的却很少！这一难以理解的现象似乎是在于，作为一个杰出的经济学家，必须具有种种才能的结合，这一点是很难能可贵的。他必须在某种程度上是个数学家，又是历史学家、政治家和哲学家。他必须精通的是把他要说的话写下来。他必须善于运用思考力，从一般原则推断出个别现象，在思想奔放中，既要触及抽象的方面，又要触及具体的方面。他必须根据过去，研究现在，推测未来。对人类性格及其风俗习惯的任何方面，他都不应当完全置之度外。他同时必须保持着既不是无所为而为之，又不是不偏不倚的态度，像个艺术家那样地头脑冷静和孤芳自赏，然而有时也必须像个政治家那样地接近尘世环境。

是不是很眼熟？你可以把其中经济换成毛泽东的军事或者编程。越是容易的入门的东西，往往是最复杂的，但是某种程度要是最简单的（远见和长期坚持）。事情往往不局限于某个领域，需要跳脱出来，主动思考联系其他的领域，扩大自己的知识体系，更好培养一个开放进步的学习系统。

习惯和自省

2019-08-24T13:54:32+00:00

Every science begins as philosophy and ends as art; it arises in hypothesis and flows into achievement.

Science without philosophy, facts without perspective and valuation, cannot save us from havoc and despair. Science gives us knowledge, but only philosophy can give us wisdom.

from 《The Story of Philosophy: The Lives and Opinions of the World’s Greatest Philosophers by Will Durant》

古希腊哲学起源是way of living，恰恰是源于平凡琐碎的现实生活，指导实践生活，一种相互作用，他们是实践中的Doer实践者。但是现代哲学听起来确实像是学术的Academic，被作为一个研究的学科，高高在上玄而又玄，犹如阳春白雪，而生活反而像是下里巴人，两者的距离确实越来越远。对于普通人更加，那时候哲人总结的东西，一个可能是语言描述不太平直（高度凝练加上每个人叙述方式的不同），一个是现代人可能并没有太多时间去沉淀下去思考为什么古人是那么想的，背景是怎么样-如果一件事情没有明显的功利性，那还不如打打游戏满足自己短期的快感Euphoria. 哲学没有快速答案也没有捷径可走，所有的事情哲学都能联系起来，但是只有你自己经世致用，知为行，行为知，才能得到自己的哲学，这个过程是无止境的，这个也是独一无二的，这个我想也是生活的乐趣所在之一吧。

笛卡尔《谈谈方法》：

真正说来，活着不研究哲学，就如同闭上两只眼不肯睁开；观看我们视觉发现的一切而得到的那种愉快，根本比不上人们凭哲学发现事物的知识而获得的那种满足；总之，我们必须研究哲学来砥砺德行、指导人生，胜过用眼睛来引导我们走路。野生的禽兽只有身体需要保护，就经常不断地从事寻求养身的食品；然而人的主要部分是心灵，就应该把主要精力放在寻求智慧上，智慧才是他真正的养料；而且我也敢断言，有很多人在这方面是不会失败的，只要他们抱着取得胜利的希望、并且知道自己能做多少就行。没有一个人的灵魂如此卑下，牢牢地固守在各种感官对象上面，不会有那么一回抛开感官对象，转过来希望取得另外一个更伟大的好东西，尽管他每每不知道这个好东西在哪里。

引入

先从1936年的《中国革命战争的战略问题》中，毛泽东总结第二次国内革命战争经验，批判“左”倾教条主义者在军事上的错误，揭示了中国革命战争的特点和规律；结合中国革命战争的经验教训，运用辩证唯物主义和历史唯物主义的基本原理，论述主观和客观、一般规律和特殊规律、战争的攻守和进退等辩证关系，提出了关于战争和战略战术方面的许多精辟见解。同时，他强调既要研究一般的战争规律，也要研究特殊的革命战争规律，更要研究更加特殊的中国革命战争规律，只有这样，才能真正懂得中国革命战争的战略战术问题。

了解事物的客观规律，必须不断的学习，同时能打破自己主观色彩上的思维束缚，同时善于总结归纳事物内部和中间的规律。好比软件开发，学习新的语言和框架（专业知识），首先是学会简单的入门hello world,到后面了解如何使用，到最后思考为什么设计（哲学思想），慢慢学多了，就能知道很多思维等构架方式都有在其他的语言和框架中类似地出现，自然也更能帮助快速了解熟悉新的语言技术了。

学习了解外部事情的运行方式（外在）自然科学，加上自我的不断总结（内在）哲学，就能构建一个适合自我的知识体系。但是两者却要是互相作用的。

牛顿曾说：“我可以计算天体运行的轨道，却无法计算人性的疯狂

这是他在南海泡沫之后说的。牛顿开始也是理性的, 就觉得股价很高，卖出了一次。结果，卖完之后股价一直在涨，挣扎许久，他还是没能忍住，重新入场了，这次高位入场之后，很快股价就开始暴跌。

自然科学可以观察计量并验证，但是人的主观是无法计算和观测。主观和客观的相互作用，密不可分。哲学是需要实践，需要联系到个人，不然就是空谈；每个人的主观都不一样，就好比每个人都拿着不一样倍数的放大镜看着不同部位和方向，看到的客观事物都会不一样。但是哲学好比是北极星，每个人都能看到，但是很少有人会留意他和自己的观察之间的关联，一旦注意到它的存在，再结合自己的观察的事情，那么必定得到到更大更深的角度，更清晰自己和事物的关系和定位。

有更多的维度的参照，可以得到更准确的定位，更大的空间认识。这个摸索过程是变化的，没有银弹，只有不断的扩展学习和总结反思，才能在不断的变化中找到一个动态的平衡点。历史上的兴衰，正是一个非常好的研究对象；政治经济人文艺术等等无一不是互相穿插互相关联，但是道理皆有相通。有人参与，就有变数，我们必须学习这些有人参与的活动的周期，来更好的了解世界也是了解人性，和了解自我。

所以周期就特别需要被研究了。

索罗斯虽然行为有争议，但是他非常有创造性的将哲学应用到金融上面值得学习。他讲金融当做他的试验田，来验证和修正自己的哲学思想。他在《金融炼金术》里提出的参与者的主动意识，影响并推动价格和市场的走向，两者是相互作用的, 互相交缠作用，好比流水和想法一样万古不息。

这本书好比是他的笔记，他如何将哲学运用到宏观经济学之上，更加不回避的提到自己的错误和修正。这本书没有提到一个明确的答案，甚至看起来有些矛盾，大部分看过想要银弹秘方的人可能要大失所望，不知所云（部分因为需要了解1970，80美国和世界的经济格局），但是正是如此，他提供的这个思辨过程才如此有价值，务实有借鉴学习。务实派明白万事的变化本质，反而乐意看到自己修正自己的过程，不管是从成功或失败之中，不固步自封。

引用其中一段话：

If we want to understand the real world, we must divert our gaze from a hypothetical final outcome and concentrate our attention on the process of change what we can observe all around us.

This will require a radical shift in our thinking. A process of change is much more difficult to understand than a static equilibrium. We shall have to revise many of our preconceived ideas about the kind of understanding that is attainable and satisfy ourselves with conclusions that are far less definite than those that economic theory sought to provide.”

需要转换思维。其实不止金融了。

一切捷克为我所用，英国自伊丽莎白一来的平衡策略，切合实际，理性分析（从典型的领土资源占据到分而治之，效率最大化），有但是，从合纵荷兰打当时第一强国西班牙，后面联合法国打的英荷战争（打败后力保荷兰），后面出力滑铁卢打败拿破仑，七年战争联合普鲁士腓特烈二世（大陆均势平衡）。一切都在动态的中找到一个平衡点，没有束缚与教条主义和经验主义，敢于自省，一种务实主义。

近几百年英国的崛起的历程，相当的有借鉴意义。相比一开始的殖民先驱强大的西班牙，当时弱势的英国，分析时局，一方面明确的采取了连横合纵，拉起尼德兰盟友拖住西班牙，一方面不断袭击骚扰西班牙的海上生命线，让本来强大的西班牙（相对英国，殖民地大多，军事强大），英国此时在局部占有了优势，同时殖民地掠夺的大量金银财宝让西班牙王室开始奢靡享受，工业和综合实力被英国赶超，最后英国打败了西班牙；后面荷兰马车夫兴起，克伦威尔英国要联合法国打击荷兰；七年战争联合支持普鲁士王国，出兵出力；后来法国大革命之后拿破仑一跃而起，横扫整个欧洲大陆，差点灭了英国，英国这时开始联合西班牙俄国，并且出钱出力帮助返工法国，最后滑铁卢打败拿破仑，恢复了欧洲均势；英国在殖民地上并没有采取西班牙葡萄牙那种直接掠夺的粗暴政策，相反建立起了大三角贸易圈（长期），允许殖民地有一定的自治，特别是印度的分而治之，几千人能管理统治几亿人的殖民地，这个比起西班牙那种粗暴的掠夺高了一个层次（短期）；

英国的定位很清晰，一个是海岛国家对海军依赖（自由贸易），一个是欧洲大陆必须保持均势。从一开始的一手坏牌逐渐打成了一手好牌，这个想起了毛泽东那个《论持久战》，在一个敌强我弱的形势下，理性客观分析优缺点，结合实际又站在一个高的角度（整体长远主要）上，在纷繁乱杂的形势中发现了问题的本质。

英国这一套是务实主义的代表了，没有固守已有的经验或意识，在不断变化的外部形势中，找到一个动态均衡点。

周期(boom and bust)

在周期开始阶段，人的预期和主动活动会提高活动的程度（士气），改造客观，同时改造了的客观反过来会加强的人的预期，这样就行了一个不断自我强化的过程，推动这事物快速发展。当这个无法持续之时，可以是人口或者资金等等，那么就开始negative enforcing阶段，不断下降的或者失败导致人气低落，反过来引起更多的失败，有如多米诺骨牌，极快猛烈地下行。

病来如山倒病去如抽丝，积累增长蓄势的过程往往比较长，但是一旦跌下来速度确实非常快。历史上国家兴衰和经济危机无一例外，这样就提供了很多有意思的对比，总结普遍和特殊规律。

然后在这个转折点出现之前，一定会有迹象；同时当意识到问题的时候，人们普遍心理为了维持已有的价值不下跌，会反而更加想投入来稳定价格（一旦价格下滑一丝丝，那么立马回放大群体效应），尽管都知道是饮鸩止渴，也别无他法，然后这个效果当然是短暂的，颓势已经无可避免，就出现了回光返照的征兆。

《大空头》里的那个一样次贷指数并没有下降，稳如泰山，尽管新闻爆出了次贷不良率的时候，那个时候这群赌博的人都快怀疑人生了，差点坚持不下去，结果这个后面就知道了。

好比Ray Dalio在《Principles》讲到的1971尼克松不挂钩美元和黄金（当然会引起美元下跌），然后政府声明保证不会贬值美元：

Over the decades since, I’ve repeatedly seen policymakers deliver such assurances immediately before currency devaluations, so I learned not to believe government policymakers when they assure you that they won’t let a currency devaluation happen. The more strongly they make those assurances, the more desperate the situation probably is, so the more likely it is that a devaluation will take place.

因为贸易战，人民币一路贬值，中间有一次政府强调不会贬值货币来平衡关税，声明之后两天有稳定小杨，随即一路跌到了7.

还有钱宝2017年8月爆出资金链有问题，张小雷直播辟谣，立马12月锒铛入狱。

从这本A Brief History of Doom: Two Hundred Years of Financial Crises 我们可以看两百年来的金融危机来更好比较就要意思了。

preconceived ideas 一些危机之前的普遍的共同意识：

1970,80年代Latin American debt crisis拉丁美洲债务危机 - “Countries don’t go bankrupt” 国家不会破产.
1990 日本泡沫经济普遍认为房地产价值和价格不会下跌 Land will not devalue.
2005 本伯南克 - We’ve never had a decline in house price on a nationwide basis.

“In the 1960s and 1970s, rapid loan growth in Japan had generally translated into commensurate GDP growth because Japan was vastly underbuilt as it emerged from the war. Workers were driven to succeed, and “bankers’ hours” had extended to the point where employees of lenders were expected to work from 8 a.m. to 9 p.m., if not later. The premium on hard work and success was so pervasive that at Fuji Bank’s centenary celebration in 1980, “employees were urged to keep working until they ‘urinated blood.’

2013, Taro Aso, the finance minister, said on Monday that the elderly should be allowed to “hurry up and die” to relieve pressure on the state to pay for their medical care.

Ben Bernanke虽然被奥巴马政府认为应对经济危机很得体, 通过FED的通用手段降息和QE直升机撒钱：

In remarks prepared for the announcement Tuesday, Obama praised Bernanke for leading the country through the meltdown and, with his expertise on the Great Depression, helping prevent a crisis rivaling that of the 1930s.

另外bail out这些罪魁祸首的institution and banks, 却没有任何对底层或者中层购房者有任何的帮助措施，（虽然衍生品很复杂理不清，公平难以保证），但是不能一刀切，象征性的照顾下也行啊，埋下了特朗普上台的种子。当然危机也让钱流向了实体，特别是硅谷。

但是看了伯南克在08年之前对Alan Greenspan格林斯潘的夸奖（05年讲座称赞是The Great Moderation), 结果发现是Great Moderation -> Greate Recession, 然后一系列的言论不无表现出“一切都在控制之内，FED掌握真理”，不要慌，及时发生的初始阶段也是拒绝接受认可危机的。

所以简单说如果我们能在boom开始时跳进去，并且在转折点之时(turnpoint)，跳出来，是不是就可以全身而退了？

难。

区别自我增强还是最后的疯狂太难了。好比我看的今日说法，我从小就爱看，里面有不少有意思的案情啊或者有法治意义的案例。里面就包括比如云南运du的，一开始敬小慎微，走小路，防止盘查，打前哨，一点点的；后来几次没被捉到，就开始大摇大摆，运的分量也是越来越大，比如住的地方就开始奢侈然后车啊房啊消费啊。我就好奇了，为什么不保持低调了，这样不是更那个安全么。还有什么套路贷啊，诈骗啊，为什么不挣了不少了，主动撤退了？

看了里面的采访，有的人也是说一开始战战兢兢，后来发现没问题就麻痹了，另外做的越大就不好处理，导致心理越来越放松（自己心理认为这么多次了，这次应该也不会有问题）却不知外界的条件可不会按你的主观来改变，你是主观放大了，但是客观事实不会跟你想的一样放大，所以风险被低估了，胆子越来越大，最后灭亡，黄粱一梦。

我后来有一次打篮球的经历让我明白了这种状态。我有一次带了一本书去打篮球（刚好看完书路过球场），然后打野球，4v4 三组轮着打，然后中间休息的时候，我就拿出书来阅读。我一开始觉得还行，慢慢我就发现我因为打球之后心跳啊出汗啊，看书上的字感觉像是一个不平整的放大镜下的字一样，跳来跳去，我很努力想专注去读懂，但是根本忘了我在看什么。

在那种欲望和状态下，理性是被人像扫把一样扫到了地毯下。swept under the carpet

这么多危机，每次过去之后，人们慢慢就觉得这次肯定是新世界新秩序，肯定不一样，不过好像每一次都一样。

自省

The changes that seem small and unimportant at first will compound into remarkable results if you’re willing to stick with them for years.

from 《Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones》

我们说习惯是需要不断的重复的，需要不断的强化，成为一种routine，才能越来越得心应手。但是当你习惯越来越久，会发现感觉效率慢慢变低了。

就需要什么了？打破这种habitual thinking.

普遍或者理所当然的事情我们需要留心，保持好奇心问为什么了？不可以因为年龄或者惯性，失去了那份天性。从大角度看，万物相通。

好比程序员，太多的理性思维，我有时候就观察到自己或别人的惯用手和走路方式，为什么不能用左手写字（费惯用手），或者为什么不能后退走路了？然后我坚持每天用左手抄写唐诗一首，然后尽量我在工作下午的时候抽空下去找一个空的地方，后退着走几圈。一方面我觉得左手写的时候，我感觉好像小时候学写字好笨拙，一方面抄写唐诗的时候不由自主的思考想象那个诗的意向，挺好的; 一方面我后退走时候，迫使我离开办公室跟自然接触，恢复感知和大自然的联系，看看天和云和竹子，也让我后面的工作有更好专注力。而且我觉得我大脑好像被按摩一样的，怪怪的。好比你不得不把自己杯子的水倒出来，从新学习熟练。

每天我都尽量抽出时间比如睡觉前来左手抄诗词，或者一些生的单词；下午或者怎么样抽出10分钟后退走几圈刚好拉伸下身体；这个事情没有啥功利性，长期坚持必定是有其中乐趣，简单的乐趣罢了，了解身体，了解大脑，也了解自己。看云卷云舒，看碧蓝灰蒙，各有其妙，也提醒自己保持初心。

惯性思维就跟人被欲望拉在两个极端一样的，很难察觉。所以要自知，要反省，要总结。

所以马卡斯奥里勒斯写了自己的日志，沉思录；索罗斯的金融炼金术更是自己试验过程的总结；一开始更多无意是对外的，是记录自己的历程。但是我们每个人都应该也有自己的日志Journal,记录自己对自己探索的过程。

要保持这个周期啊，就需要平衡，需要反思这种习惯性思维，才能保持一个开放并且澄明的心态，更好应对变化的外部。

“You have power over your mind — not outside events. Realize this, and you will find strength.” (Meditations, Marcus Aurelius)

没有反思反省的过程，就不是一个双向良性的自我增强的过程；停止了就会走向反面，切不可范经验主义错误，更不能因为成功而沾沾自喜骄傲膨胀，但是难啊，反人性啊。毕竟谁愿意一生保持敬畏，过危桥一般了，那不就是圣人么？

我想我们只能不断的通过重复的习惯来建立一套体系框架，至少让犯错的时候代价小点，或者让我们能够离本心更近点。

致虚极，守静笃，万物并作，吾以观其复。

夫物芸芸，各复归其根。

归根曰静，是谓复命。

复命曰常，知常曰明，不知常，妄作，凶。

知常容，容乃公，公乃全，全乃天，天乃道，道乃久，没身不殆。

这个体系又要保持，又要反思然后拥抱变化，这个太难了。。。

这个度和平衡如何把握，一生都没有明确答案吧, 但是我们一生不就是在不断寻找这个动态平衡点么？

反过来，探索这个过程，应该挺有意思的。

最后附上最近看到的Ryan Holiday的一篇文章：Why You Should Study Philosophy

沉思录Meditation

2019-05-26T13:54:32+00:00

引入

先引用2500年轻苏格拉底两段话:

“I honor and love you: but why do you who are citizens of the great and mighty nation care so much about laying up the greatest amount of money and honor And reputation, and so little amount wisdom and truth and the greatest improvement of the soul? Re you not ashamed of these?… I do nothing but go about persuading you all, not to take thought for your persons and your properties, but first and chiefly to care about the greatest improvement of the soul. I tell you that virtue is not given by more, but that from virtue comes money and every other good of man. ”

“Fellow citizens, why do you turn and scrape every stone to gather wealth, and, yet, take so little care of your own children, to whom one day you must relinquish all.”

本杰明富兰克林在20岁的时候给自己定的目标：努力达到完美品德moral perfection.

I conceiv’d the bold and arduous project of arriving at moral perfection. I wish’d to live without committing any fault at any time; I would conquer all that either natural inclination, custom, or company might lead me into.

达芬奇：

Who sows virtue reaps honor.

我一直在想如何入手些这篇文章，因为要表达的东西感觉有点杂乱。

有外到内

“Acres of Diamonds,” by Russell H. Conwell, is a story with an ageless moral of a man who dreamed of finding riches and fruitlessly traveled the world for decades in search of a diamond mine. Exhausted, tired, broke and discouraged, he returned home only to discover that the diamonds he sought were right under his nose—at the farm he had sold and left behind.

Acres of Diamonds by Russell H. Conwell - Full Audiobook youtube

大意：讲了一个农夫梦想更大的财富离开家，四处寻找黄金，结果多少年都一无所获没找到，而且自己累死累活精疲力尽。然后他回家之后却发现就在他鼻尖下的农场下面就有钻石。

如果看过这段的，可能会联想到《牧羊少年奇幻之旅》最后的宝藏居然是在自己原来放羊的那个教堂废墟下面。

我想这个过程是需要折腾一番的，我不知道钻石的英亩中的农夫如果不出去一趟，是否一生能否意识到自己的土地下就有钻石；亦或是牧羊少年不是出去寻宝之旅遇到的各种奇遇，后面的宝藏原来在自己老地方的这个扣题会不会有意义。

斯托葛派Marcus Aurelius:

“Dig inside yourself. Inside their is a spring of goodness waiting to gush at any moment, if you keep digging.”

毫无疑问：由外往内看，是需要折腾的。

Rumi鲁米：

We carry inside us the wonders we seek outside us

《无门关》禅偈一首：

佛在灵山莫远求,灵山只在汝心头
人人有个灵山塔,好向灵山塔下修

无名氏的非常一首：

尽日寻春不见春，芒鞋踏遍陇头云。
归来笑拈梅花嗅，春在枝头已十分。

生活了没有经历或者遭受重大打击或挫折，有时候这些看起来像是像是无病呻吟或是玄而又玄，虚无缥缈，远离人间烟火，因为只有当完全接受(surrender&forgivenessThe Power of Now)现实，人才会重新审视自己的内心，也才会明白背后的深远意义。

思考自己已有的价值观或者跳脱出来得到开悟。但是从何开始？

自我觉察Self-Awareness

自我觉察是第一步。

觉察什么了？觉察自我形象, 情感emotion, 身体感知(body sensation)和自我(ego)。

先讲一个（Psycho Cybernetics，虽然翻译的书名看起来很扯淡，但是考虑这事一本1960年代的书，还是当时非常有前瞻性的。作为一名知名整容外科医生，马尔茨博士发现，许多患者尽管获得了新的面容，但其内心的痛苦与缺乏安全感仍然影响着他们的人生。

有人整容了，但是内心里还是得不到自信；有些人来整容的人，真正可能不是需要整容，而是心理需要咨询。想想现在各种整容广告（特别在湖南什么都市频道等，各种啥亚美等，宣传遇见更美的自己？），我在想有没有整容医院开设一个心理咨询(但是商业上不挣钱，估计不是这个年代的风格），其实大部分整容的人应该是心理上的问题更多，至少大部分他们还纠结在外在的自我（self-image), 在他们心中等同于内心的self-image自我形象。同理的还有学生，成绩不好，可能根源在于他的认知，太多外在的干扰，让这个年纪还未有完整自我形象的孩子失去了方向，只能是抱着自暴自弃亦，所以很多时候都可以听到：你看我就是不行，没天赋。

I是什么？我是什么？这个年代我想大部分都感觉到定义一个”我“，更多是从外在的金钱，名誉名声，权力等等。别人怎么评价我？怎么看我？跟别人对比？从而产生很多负面的甚至破坏性的情绪，如果管控不好，必然会产生不好的结果。特别是在当今这个信息化的年代。

人类的进化史，从原始人在非洲草原，正面遇到老虎(stimulus刺激物），注意到它，解释(interprete)为一种威胁，这个时候必须毫厘之间决定是对抗还是逃跑(fight or flight), 本能的我们身体会加速肾上腺素到血液中，关闭消化系统，心脏跳动加快，身体肌肉锁紧，随时最好寻找安全的准备。人类百万年一直在寻求延续，这种机制已经形成模式深深耕种在潜意识了，只要一旦特定形势出发或者说共振，大脑会跳过理性思维（分毫之间），直接启动潜意识了形成的模式，因为这是最简单也是最高效的求生方式。

刺激物 -> 解释 -> 本能的情绪或者身体反应

自我是一种防卫机制，一种经验或者生活经历规则形成了一套判断系统。但是有时候也无法区分现实和EGO的想象。当刺激物出现，自我这一套在潜意识里被触发，很有可能会直接触发，不经过理性思维。

以我来看，跟别人探讨问题时，可能因为某个人的之前的一些行为让我看不惯或者我认为他有故意针对我的意思，那么他语调的一个细微变化（刺激物），就很有可能出发我潜意识的自我防卫-这个人好像对我有威胁或者是名声或者是能力上挑战，继而可能我可能失去工作失去生存，那么我可能立马就开始有情绪化的表现，比如脸变红，身体不由自己的发抖收缩，语调更高，而且仔细看说的话也越来越脱离事实，好像变成了个人之见的针对。但是事后下来，又觉得自己好像反应过度，不知道自己为什么当时情绪失控，为什么理性好像没有了，变了一个人。

Plato柏拉图：

Human behavior flows from three main sources: desire, emotion, and knowledge.

Emotional Intelligence情绪智力 - 情绪管控

Stoicism teaches the development of self-control and fortitude as a means of overcoming destructive emotions;

斯多葛学派（Stocism)核心的教学之一也是让大家发展自我控制来克服破坏性的情绪。 ps: 心理学上负面的情绪对人的冲击是同等正面情绪的两倍，所以两倍的痛苦=一倍的喜悦。也要说的是其实太positive的情绪也有坏处，两者都得小心对待和认识。

到了当今，结合心理学，和大脑的研究，也让我们更多了解到潜意识，理性思维和情绪怎么工作的。

自我 - 喜欢安逸，短期愉悦需求，其实是我们大脑创造出来的一个幻想中的自己。很多时候，我们也是这么觉得，不会注意到它的存在。但是长期来看，它让我们模糊了视线，失去了自我。它需要外在的输入来feed饲养，更多时候是外在的物质的有形或无形的。

好比什么了？我想应该是毛泽东笔下的纸老虎。更贴实际的便是熊孩子。一个熊孩子的养成记，必定是开始的时候，孩子哭着闹着要什么东西，父母可能能拒绝一两次，但是架不住啊，心想赶紧结束这个吵闹，真是烦的很，与其坚持自己的原则（长期，正确的事），还不如短期弄一个清净。所以就答应了孩子的要求。结果孩子果然拿到糖果就不哭闹了，但是下一次了？这好比不断透视饲养一只老虎，你越怕他，越给它投食，它后面就越加肆无忌惮。恶性循环.

所以我觉得需要认识到这个自我的存在。同时，在解释和反应之中加一个缓冲区间

The Stoic tells himself that although the situation may appear frightening, the truly important thing in life is how he chooses to respond.

认识到现实和解读的存在，认识到当情绪的变动时，身体的感知变化时，有意识的觉察到这些，而不是立马给出本能的反应。让你的情绪和你身体的感知成为你的朋友，你可以读出也可以读懂它的信息，然后退一步，给一个缓冲的空间，比如”ok,我注意到我身体好像有点发烫，牙齿咬得有点紧“，好像是有点危险的信号，需要理性的介入，来评估是否是真实的情况了。我注意到我有一个想法， ”我觉得他在针对我“ 而不是”他在针对我“。引入这个缓冲空间，这个理性的空挡，可以问一些”他说的是不是事实了，是不是可以让我变得更好“。

一点点的open-minded, non-bias的理性思维，会让整个后面的反应变得更好。

okay,那么有哪些实际点办法来管控情绪了，除了听别人说的，因为我的有办法能够把这个引入到我的日常生活之中啊。

是的，我可以快点提一下我的方式：冥想和深呼吸，散步，太极拳。后面会继续谈到。

信念Conviction&Faith

Roy T. Bennett

“Don’t let the expectations and opinions of other people affect your decisions. It’s your life, not theirs. Do what matters most to you; do what makes you feel alive and happy. Don’t let the expectations and ideas of others limit who you are. If you let others tell you who you are, you are living their reality — not yours. There is more to life than pleasing people. There is much more to life than following others’ prescribed path. There is so much more to life than what you experience right now. You need to decide who you are for yourself. Become a whole being. Adventure.”

如果这个自我不是真的自我，那我是谁？怎么定义我之为我？

当我踏进了30岁的时候，感觉好像有点紧迫了，开始感觉到了那种焦虑，一种好像生活越来越易碎也来越没有定数的感觉。回想小时候，最怕想到死，想到浩瀚宇宙，想到死后什么都没了，没”我“了？

我想着从东西方的先哲那里应该可以得到很多启示吧，如果我能，并且融入到自己的生活中，即使是卑微的不起眼的俗人一个，那也应该是一件很有意思的事情，起码我想体验这个过程。

活出信念，应该是一种啥感受了？苏格拉底死于信念到近代李小龙将东西方哲学揉搓在了截拳道了。我好奇如何在一种未知不确定的情况下仍然坚持相信自己的内心，不依赖于外在，不从众，活出自我。

要知道我们人是群居性动物。古代科学不发达之时遇到未知之事，人们心里恐慌只能借助于外在的所谓神，于是有了祭祀，各个文明的诞生都有类似的致意，甚至以牺牲同类人的生命来祭祀。从众心理让个人可以有了借口，摆脱了”自己需要为自己行为负责“的包袱，就像羊群，面对未知，至少我跟大部分人是一样的，责任也不会全在我一个人身上。why not?

temperament气质：你不需要因为跟大众站一起而感觉高兴兴奋抑或是站在大众对面而高兴兴奋。

Neithe drive great pleasure from being with crowd nor against crowd

你需要找一些比自己更大的东西，超越个体的东西。是什么了？

不要去思考，去感受 Don’t think, feel & experience

这是李小龙在电影里的一个片段。这段的说的是李小龙让学徒不要只看着手指，否则会失去明月的光辉。

要理解这段话，我觉得的理解李小龙的创建截拳道的背景和他的人生经历。他是学哲学的，应该说博览群书，东西方哲学都是广泛涉猎，也包括东方的禅和道，并结合武术的实战特点，独特的创造性的创建了截拳道。与其说他是武术家，我觉得他哲学深度更加令人痴迷。

BruceLee Podcast是一个很好的资源。还有书《The Warrior Within: The Philosophies of Bruce Lee》

月亮不是一般的月亮。唐朝寒山的《吾心思秋月》：

吾心思秋月，碧潭清皎洁。无物堪比伦，叫我如何说！

道家开篇：道，可道，非常道。名，可名，非常名。可以说的道不是道？非也？

佛家里有捏花一笑，也有德山棒、临济喝，更多有听到般若波罗蜜。智慧的彼岸，它人之口说出来的东西是它人的领悟，只是一个路标路牌，标注了方向。经他之口必然是他的领悟，但是没经你的口没经你的心，就不是你的东西，也不是道。道，需要自己走路渡河出来的，重在身体力行。悟靠自己，他人替代不了。

月亮代表悟道的心，澄明，没有杂念，清澈宁静。当然也是个比喻啦，但是你要知道它在哪里。老是听到：禅不可说，不可说！这句话显得有点玄乎了，但是跟道一样，必须是你自己经历体验（Experience). ”如人饮水，冷暖自知“

没有体验经历，让人如何悟，读起来不就觉得玄而又玄，形而上学；更别提，那种更高层次的精神共鸣，宇宙自然朴素存在的真理。

体验经历，抛弃固有的偏见，不管好坏（也无所谓好坏），从其中学习。这让我想起了Ray Dalio在《原则》提到的一个循环:

** Experience + Failure/Success + Reflection(examine emotion/feeling honestly by fully accepting reality) = Next Level **

陆桴亭《思辨录辑要》卷三云：“凡体验有所得处，皆是悟。只是古人不唤作悟，唤作物格知至，古人把此个境界看作平常。”又云：“人性中皆有悟，必功夫不断，悟头始出。如石中皆有火，必敲击不已，火光始现，然得火不难，得火之后，须乘之以艾，继之以油，然后火可不灭。故悟亦必继之以躬行力学。

我们需要有效的experience, 必须抛开一些理所当然的主观上的东西，好与不好的体验，都是体验，现实是现实，只是我们主观的解释让我们有了倾向性，可能影响我们看到本原本质，我们需要完全接纳accept现实，并且从中reflection反思总结学习，然后进入下一个体验循环。

这个过程中，你无法欺骗的是你自己，可能需要点时间来沉淀情绪心理，but it is okay, take easy,都不是圣人，不必苛求，但是我们可以从情绪感知中读懂真实的自我，来帮我们正视现实.

“不识庐山真面目，只缘身在此山中”

有时候需要跳出来，重新审视我们的认知和行为，才不至于在局部迷失自己。

事实上我想继续深入探讨我自己的想法，但是又觉得在这之前可以先看看近代的一个先驱的体验。

富兰克林

《Benjamin Franklin: An American Life》by Walter Isaacson, 对了Walter Isaacson写了不少的传记啊，达芬奇的也是很不错。

富兰克林真是个多才多艺之人，科学发明等等很多身份，他很多的发明并没有申请专利为一己之私，而是开放了给了公众，他知道知识不是一个人的专有，而是所有人共有的。他有他的十三个名言（Franklin’s 13 virtues）：

Temperance: Eat not to dullness; drink not to elevation. 节制：食不可过饱，饮不得过量。 Silence: Speak not but what may benefit others or yourself; avoid trifling conversation. 缄默：避免无聊闲扯，言谈必须对人有益。

Order: Let all your things have their places; let each part of your business have its time. 秩序：生活物品要放置有序，工作时间要合理安排。

Resolution: Resolve to perform what you ought; perform without fail what you resolve. 决心：要做之事就下决心去做，决心做的事一定要完成。

Frugality: Make no expense but to do good to others or yourself; i.e., waste nothing. 节俭：不得浪费，任何花费都要有益，不论是于人于己。

Industry: Lose no time; be always employed in something useful; cut off all unnecessary actions. 勤勉：珍惜每一刻时间，去除一切不必要之举，勤做有益之事。

Sincerity: Use no hurtful deceit; think innocently and justly, and, if you speak, speak accordingly. 真诚：不损害他人，不使用欺骗手段。考虑事情要公正合理，说话要依据真实情况。

Justice: Wrong none by doing injuries, or omitting the benefits that are your duty. 正义：不提损人利己，履行应尽的义务。

Moderation: Avoid extremes; forbear resenting injuries so much as you think they deserve. 中庸：避免任何极端倾向，尽量克制报复心理。

Cleanliness: Tolerate no uncleanliness in body, cloths, or habitation. 清洁：身体、衣着和居所要力求清洁。

Tranquility: Be not disturbed at trifles, or at accidents common or unavoidable. 平静：戒除不必要的烦恼。也就是指那些琐事、常见的和不可避免的不顺利的事情。

Chastity: Rarely use venery but for health or offspring, never to dullness, weakness, or the injury of your own or another’s peace or reputation. 贞节：决不使身体虚弱，生活贫乏，除非为了健康或后代的需要。不可损坏自己或他人的声誉或者安宁。

Humility: Imitate Jesus and Socrates. 谦逊：以耶稣和苏格拉底为榜样。

富兰克林这十三条应该说非常贴近日常了。

他还会在每天早上和晚上问自己我将会做什么好的和我做过哪些好的这些问题，这是每日三省吾身么？

道家，禅，苏格拉底，亚里士多德，斯托葛派这些不同年代不同地域的学说，好像有一些共通的之处：比如如何寻找快乐，关于个人如何更好认识自己，如何跟宇宙自然打交道，最重要的是平衡（我的看法）。

上面十三条不一定每条都贴切，但是有些启发性的意思，结合下面的。

小道修身

先看斯托葛派定义：

Stoicism is a philosophy of personal ethics informed by its system of logic and its views on the natural world. According to its teachings, as social beings, the path to eudaimonia (happiness) for humans is found in accepting the moment as it presents itself, by not allowing oneself to be controlled by the desire for pleasure or fear of pain, by using one’s mind to understand the world and to do one’s part in nature’s plan, and by working together and treating others fairly and justly.

Nature自然 - 好像跟道德经里的道的起源 present moment（当下）- 跟禅宗里活下当下有点类似了。

理解世界，理解万物运行的规律，可以帮助我们接受现在当下，不被情绪欲望掌控，遵从初心内心本心的指引。

马可·奥勒留（Marcus Aurelius） & 爱比克泰德（Epictetus），一位是罗马皇帝，一位是奴隶，尽管悬殊的身份，结果都成了古罗马著名的哲学家。马可·奥勒留身为皇帝，一切欲望都可以满足，因为他的权力，也使得他成为应该是最孤独的人，他写沉思录并没有打算公开，只是写给自己，一种纾解孤独的方式。面对如此巨大的欲望诱惑（想象下），他却没有放纵，反倒了引申出了跟他孤独相对高度的人生思考：如何更好的自我控制。

The measure of a man is what he does with power.

道德经里面讲了很多有大道，当然也有个人相关的道。修身的道，道家，禅和斯托噶都殊途同归。

比如追求happiness幸福，是每个人的都想的。但是真正的幸福是啥了？

物质，欲望等如果不加注意，很容易陷入它们的圈套中，迷失其中。不是说不好，但是需要清楚这些东西不定义自己，不能过度执着其中而不能自拔，继而放纵过度。

道德经：

五色令人目盲，五音令人耳聋，五味令人口爽

大部分情况下，人都像是骑驴找驴或者说国外的说法是chase the wrong rabbit. 物质欲望了有了正确的认识，跟刀一样，是无害的，甚至可以发挥他的优势；但是没有正确的认识心态，短暂的极度的满足感之后接着的是更巨大的空虚感，需要更大更强劲的刺激物，到头来可能还是黄粱一梦，或者陷入更大的循环中。

安贫乐道固然是好，但是有很多的物质拥有也可以乐道，取决于还是自我。没有这些物质或者外在的拥有，有或没有，我都是我自己，我可以不用这些东西来定义我自己，我完全接受当下的自我，这是真我本色。

Honstly express yourself. - Bruce Lee

我想这是一种个人的道吧。

很多施工工地的护档都有很多标语，其中就有”静以修身俭以养德“。我非常同意静以修身，静能生慧，特别是你有体验，再静再定，则是美妙无比；俭以养德，同样的，富也可以养德。

古人的，中庸之道，平衡之道真是大智慧。可能我们教材读书时过多的讲了儒家孔子的东西，对于老庄的东西没有提的太多，当然也能理解如果让白纸一样的没有实际入世经历的学生来体悟这些东西，好比无根之水。到了一定年纪，一定经历，重读道德经，真是发人深省，赞叹不已。现代社会以来，这些古老的美玉如同蒙尘一般，被世人遗忘，大家都蜂拥而向摩拜西方的学说，却不曾发现这个”灵山“就在自己脚下。

我时常带着这本口袋书放书包里，有时候有空拿出来看看翻翻，不一定有什么特意，随手一翻，然后不去想，就是感受。

这本书严重被低估，正如我第一次将信将疑的准备读时，好像跟道德有关，别人说你是不是”缺德“，哈哈，可能是吧。正如我说，我注意到了我这个偏见，心想，but bear with me，开放的心态去看看吧，果然这本书真的蕴藏了很多哲理，瞬间解答了我很多的疑惑。

道，是万事万物运行的规律，变与不变，阴阳转换，他是风一样，无孔不入，大到宇宙，小到尘埃，都可以走到它的身影。

大自然更是充满了它的存在，如果你仔细观察。

不偏不倚，自然能看的客观，不会片面，但有任何妄念，自然会让你脱离中间，失去看透全局之势；其次，整个宇宙或者大自然，一切都在平衡之中达到和谐，但是任何偏颇，都是差之千里。阴阳互换，循环中自有这个锚点，万事在平衡中打破，打破中平衡；自负自傲则必有大败，自卑菲薄则难以起誓，但是如果知道这个中的存在，则必定在自傲之时，觉察自我，谦卑才得持久；在自卑之时，觉察自我，只月亏则盈，不必执念于一时，更不必妄自菲薄；

This, too, shall pass

中庸平衡 - moderate

将欲歙之，必固张之。将欲弱之，必固强之。将欲废之，裕固居之。将欲取之，必与固之。

上帝想让谁灭亡，必定先让其疯狂。如果想要毁灭某一个东西，先推举这一个东西，让他到达顶峰的时候，他自然会走向灭亡。

柏拉图：

There is in every one of us, even those who seem to be most moderate, a type of desire that is terrible, wild, and lawless

对于人来说特别，当他情绪高涨，自认为天下无敌（EGO），感觉良好，他会忽视规律或者道，他的EGO自我让其凌驾于客观事实，为什么了？感觉太好了，这种快感(euphoria), 短期极度充满自我，想气球一样，飘了起来，脱离了地面。好比到了一定高度遇到冷空气，不然破气下落摔下来。他让丧失了理性的判断，overriding复写了你的大脑程序，某种程度上控制了你；人一旦产生妄想，就会违背自然规律而恣意妄为。

Lord give me humility, especially when I think I need it the least.

当你遇到挫折逆境，有时候你会陷入到负面的情绪之中。自我ego会极力打击你，心理学上，它会让你不是就事论事，或者站在长远角度来看说这没什么大不了的，而是让你在短暂之中强烈的否定自己，”我不行，我肯定做不好，我是个傻子“，最重要的是它会联想起你潜意识里的积累的负面经历，一并累加叠加，让你一下子无法区分失控，陷入强烈的自责自我放弃等破坏性的情绪；

Rumi鲁米:

These pains you feel are messengers. Listen to them.

禅诗王维：

行到水穷处，坐看云起时

这个时候就需要一些缓冲，一些耐心patience, 定能生慧嘛。

这好比一个弹簧，我们需要观察我们是不是在两个极端，或者至少意识到，自我觉察。大道在平衡循环，物极必反，正确认识自己，观察情绪和身体反应，给自己一点rational thinking理性思维的空间，我们就能用道平衡自己。

不以物喜不以己悲，可能太难了，但是我们可以做到觉察，做到中庸之道，跳脱出这个圈。

六祖慧能和神秀的偈有点意思：

神秀：

身是菩提树，心如明镜台，
时时勤拂拭，莫使惹尘埃

惠能：

菩提本无树，明镜亦非台,
本来无一物，何处惹尘埃

咱这种凡人，无法做到惠能的那种悟境，但是大部分人都可以学习神秀偈中的渐修。禅宗南北之争，渐修和顿悟本可相辅相成。

时时勤拂拭，莫使惹尘埃

富兰克林关于pride和humility一生都在对抗之中，不晓得如何处理。

今天应该是NBA东决的Game6 elimination淘汰赛猛龙vs雄鹿，我看了一上午，所以导致现在还在写这篇博客，猛龙赢了东决，队史24年来第一次进入final总决赛。赛后东决典礼上，TNT主持人Ernie Johnson采访莱纳德kawhi leonard,关于猛龙第三节末端落后15分，护士（范乔丹挡子弹的那位）教练说：

Nurse: Even keel; when things going well, dont’ get too carried away; when they down, don’t get too down.

Kaiwhi: enjoyed the moment and went there hard playing each possesion.

Toronto Raptors Trophy Presentation Ceremony - 2019 Eastern Conference Finals Champions

Even keel 哈哈，大道相通, 万物为一物。当你打得好时，别太得意忘形；当你打得差时，别太失落；莱纳德完美说的到他当时的想法：享受当下，我就是打球，一球一球拼。(威少躺枪）

变化不变，完美不完美 - 无为&活在当下

The universe is change; our life is what our thoughts make it.” Marcus Aurelius

John Allen Paulos:

“Uncertainty is the only certainty there is, and knowing how to live with insecurity is the only security.”

我之前一个疑惑是对未知对变化的忧虑。天下大势，分就比合，合久必分

反者道之动。弱者道之用。天下万物生于有，有生于无。

一切接在变化当中，唯一不变的是变，变乃是不变。稳定会归于不稳定，不稳定会归于稳定。不完美才是完美，因为有了变化，有了不足，有了变数，才有动。活在当下，顺其自然，接受不确定的因素，并且正确看待它（没有不确定，就失去体验的乐趣，这不是道），顺应规律，最好自己能控制的，让不受控制的顺其自然，平和的心态去欣赏。

这也是无为的核心思想。无为并非被动式的逃避问题或者消极厌世，而是你做好的功课，了解观察反思总结事情运行的规律，到了该行动的时候，主动出击，倾尽全力，不纠结于结果；完事，平和的心态接受任何的结果。先思考你能给予什么，而不是先思考我能获得什么。

Stoic 爱比克泰德Epictetus:

Man is troubled not by events, but by the meaning he gives them. … What, then, is to be done? To make the best of what is in our power, and take the rest as it naturally happens.

正如禅里的：因上努力,果上随缘。

比如打篮球，你平时努力训练，看录像研究对手，到了比赛上，倾尽全力，不去思考结果，那么就是最好的果。但是很多人没有这么去理解，他们不努力然后求果，或者太多人注重在结果(attached)，却没有努力在过程。

太多时候，我们越想得到，实际了越加得不到；反而我们放下了，事情往往水到渠成。

The excessive increase in anything causes a reaction in the opposite direction.

记住我们需要完全接受当下，并不是消极不进取，相反一个人如果能平和接受当下的情况，那他就能不畏首畏尾患得患失，全力以赴(commitment&unattached)，因为每个时候他都能用心体验(experience)，平和接受，不管结果如何。

苏格拉底：

“He who is not contented with what he has, would not be contented with what he would like to have.”

水 Humble&Humility - Be Water, My friend

曲则全，枉则直，洼则盈，敝则新，少则得，多则惑。是以圣人抱一为天下式。不自见，故明；不自是，故彰，不自伐，故有功；不自矜，故长。夫唯不争，故天下莫能与之争。

苏格拉底说I know that I know nothing. 人在自然宇宙面前太渺小了, 知道大道的存在，必须跳出个人的偏见，以及大的胸怀和坦诚来处世。同时人作为人，正如傲慢对自卑，优越感对卑微感一样，需要明白两者互相转换，你对他们飞扬跋扈，必然会碰到人对你飞扬跋扈；抛开虚无空实对立，人必须平和平等以心对另外一个同等的人，了解他们的存在，敬畏他们的道，不能轻易被外在表现蒙蔽。

水善利万物而不争，处众人之所恶，故几于道。居，善地；心，善渊；与，善仁；言，善信；正，善治；事，善能；动，善时。夫唯不争，故无尤。

水德是近于道.

李小龙的一段经典采访Bruce lee the lost Interview 中的台词：

Empty your mind, be formless. Shapeless, like water. If you put water into a cup, it becomes the cup. You put water into a bottle and it becomes the bottle. You put it in a teapot it becomes the teapot. Now, water can flow or it can crash. Be water my friend.

道德经中：

天下莫柔弱于水，而攻坚强者，莫之能胜

理解这段话，需要理解这段话之前的采访；他强调平时要训练，练到一种肌肉记忆，想打哪打哪，心到手到，分毫不差，这个非常重要。然后就是下面这段话，你要清空思维，（实则忘记所有招式），就水流一样，根据形势自动触发肌肉记忆，（这个过程不需要理性思维介入，因为已经养成了一种直觉intuition),是最快也是最灵活的，所有的招式有潜意识里刻录的模式pattern自动做出肌肉反应，达到一种无我无形的状态，与道合一。（咋感觉有点像牧羊少年奇幻之旅中那个宇宙的语言）

刻苦训练 + 潜意识 = 直觉

需要长期的培养，时间是催化剂，没有捷径。没有培养训练过的直觉是瞎蒙；

open-mindedness 开放

Ignorance, the root and stem of every evil.

Ray Dalio 《原则Principles: Life and Work》提到的开放思维的原则（Be Radically Open-minded)。当你有了固有的偏见，你就很难去尝试接受新的东西；当你习惯了安稳，自然就很难去接受变化；人成长之间慢慢形成了自己的三观经验规则，慢慢会局限自己的认知；实则到了一定程度，却要反向为之，打破自己亲手建立起来的壁垒，重新建立。

要进化evolve, 需要这样。开放的心态在于认识到他们知道自己很多都不知道，如果事实可以帮助他们了解更多，他们不介意承认自己的错误，无惧别人的不屑亦或是嘲笑的眼光，因为他们知道自己想要什么。而封闭心态，强调防守，从自我（ego)上拒绝承认潜在的事实，因为它可能很痛，逃避很可耻但是很有用，他们在意别人的目光，哪怕事实对他成长有帮助。

所以碰到封闭之时，有时候可以问问： “这个对我有没有帮助了？能不能帮助提高我自己了？”

实际上，荀子讲的非常好：

以仁心说,以学心听,以公心辩

我平时也有新来的程序员，尽管我强调让他们平时可以多积累下英语，多写博客，记录下自己的心得，但是很多时候他们会问“我写的博客太入门水平，不够台面，甚至丢人，不太想写了”。我说你写博客，是帮助自己理清知识点构建知识体系非常有帮助的一个习惯，你写的是你的思路，你的想法，你不要怕别人怎么看，尽量honstly express yourself, 别人看的是你主动思考的过程。我相信任何一个成熟体面的程序员是不会去嘲笑别人的东西，反而会欣赏写作者的用心和努力。至少我会非常肯定非常欣赏。

罗马非一日建成，这中间形成良好的习惯则是罗马更伟岸的存在。

开放有时候就是抛开自己偏见，比如21天的计划，或是当我开始学习冥想的时候（2018/10开始坚持到现在）。当时开始的时候，每天5分钟，盘腿坐在那里，闭上眼，思维简直是一批野马，各种感受扑面而来，那种一刻都做不下去的心里，让人真是如坐针毡。虽然我刻意避开人群，不想让人觉得自己神神叨叨傻傻比比的，还是觉得自己在搞毛，不晓得自己在干什么，还要体验这么难受的感觉。过了一个星期，我感觉有点坚持不下去，我注意到我有点搞不动，你能想象你不带功利性的做一件事情，好像一件没有啥目的意义的事情？

我的自我察觉提醒我，也许先抛开这些难受等再试试，毕竟难受跟愉快一样，本身就是一种感受，也许我太过倾向我喜欢的，让原本也许没那么痛苦的东西被我放大了很多倍。那就试试咯，这一下来，就成为了我的一个生活习惯，就像刷牙洗脸洗澡，没啥功利性，你也不会花一秒钟思考为什么要这么做，然而你就这么做了，每天25-30分钟，两次，当然有时候有事情或者什么只有15分钟，没关系，别那么苛刻嘛。

有时候在机场，中巴车，出租车或者等红绿灯，有点空隙的话，我尽量尝试做一些深呼吸或者坐姿或者盘资的冥想。我觉得我做这个事情本身没有什么功利性，但是它却实实在在给我带来了很大的改变，后面再说。

冥想之前记得拉伸身体。

开放不等于胡乱接受，全盘接受，正如前面的循环中提到的，你还需要reflection&learning&feel你需要总结学习反思在你的理解上，该摒弃的摒弃，该接受的接受，至少给你开了不一样的窗户看到不一样的风景。这个建立在自己的思维体系之上，而且也需要迭代（软件开发中的敏捷开发一样Agile），定期做retrospective(回顾会议）等等。

Simplicity (返朴归真)

大道至简，返朴归真。

less is more

complexity to simplicity

single task

万变不离其宗, 我们需要观察现象的得到规律，这个规律简单但是长久。简单(simplicity) + 时间(time) = 长久（longevity）从看山是山，到看山是山，境界不一样，从简单到复杂，再从复杂到简单。

禅语说： 吃饭的时候吃饭睡觉的时候睡觉

然后牧羊少年奇幻之旅里说： 不要失去耐心，就像赶驼人所说，该吃饭时吃饭，该走路时走路。

有点邪了，牧羊少年奇幻之旅这本带有宗教色彩的书，居然跟禅里的活在当下这么类似，而且是意思非常类似。

有时候使我们attach投射了太多自己的欲望，让原本简单事情变得复杂，也带来了很多困扰，初心蒙尘；我们需要个度，需要一个空间，一个静的空间，一个退一步的空挡，来看看这个大的big picture.

太多的并发编程，你知道的带来的耗损和维护的复杂度，也是mind-blowing. 有时候生活需要做减法，简单化，我一次只做一件事情，但是我能保持高度的专注，不需要频繁切换context，但是我却比我原先认为的diversify或者parallel并发效率反而更高。 (single point focus)

有时候编程也是像生活。

手捏青苗种福田，低头便见水中天。
六根清净方成稻，后退原来是向前

in long run:

Simplicity is the ultimate sophistication.

乔布斯on冥想(Steve jobs once told to Issacson about why he meditate.)：

“If you just sit and observe, you will see how restless your mind is,” If you try to calm it, it only makes things worse, but over time it does calm, and when it does, there’s room to hear more subtle things - that’s when your intuition starts to blossom and you start to see things more clearly and be in the present more. Your mind just slows down, and you see a tremendous expanse in the moment. You see so much more than you could see before. It’s a discipline; you have to practice it.”

冥想是一种方式，好比金刚狼休·杰克曼（Hugh Jackman）在Nothing has ever opened my eyes like meditation,” says Hugh Jackman谈到的:

让你接近更简单本真的自我：

“I would describe it more as a revealing of consciousness—that through meditation on a daily basis, I get to strip away the masks that we build—that I build for myself, small and large—to reach more of a feeling of my true self: Oh, this is who I really am. This is how I can experience life. Oh, I see. It’s just something simpler, finer and more powerful. “

关于冥想让大脑让思维澄清：

“I like the analogy of a glass of water…when you first pour it it’s cloudy. When you’re stressed, that’s what your mind is like, it’s kinda cloudy. And after I meditate all that sinks to the bottom and the water is clear and the energy is finer, and the decisions you make are more authentic, and I think you’re more economical with your energy, with your time, in every way.You’re more able to listen to other people.

我无法再同意，休·杰克曼看了之前他很早很早时候的采访，他在spritual这个上面真的还是有独特见解。

但是我提醒冥想必须不能带有功利性，正如《禅者的初心 zen mind beginner’s mind》所说，同时你必须把它当做一个日常daily route惯例来做，临时抱佛脚或者像我爸妈一样高考前一天去求神拜佛，大概率是行不通的。（虽然我高考成绩还不错）

冥想也让我对编程有了新的感知，至少我的专注力是提高了不少。

道 - 万物为一物，大道相通

道，在万物之中，特别是我们身边的大自然。朴素中不乏睿智，激荡中不乏静定。

Macus Aurlius:

“Every living organism is fulfilled when it follows the right path for its own nature.”

每个人都有一个小宇宙一个自己的nature, 跟外面大道对应。我们应该想办法让两者联系起来，明白小道和大道是一体，可以借鉴，和谐并且共鸣来激发创造力。（Origination&Creation)

Virtue consists in a will that is in agreement with Nature.

大道在大自然哪里了？四季变换，春是播种，秋时收获，自古皆是此理。它充斥在宇宙了，每一个地方，是万物，也是一物。

道生一，一生二，二生三，三生万物

万物都有互相关联，并不是独立孤立存在。我们的狭隘有时候会让我们看不到那个闪光。

李小龙Oneness：

You know what I want to think of myself? As a human being. Because, I mean I don’t want to be like “As Confucius say,” but under the sky, under the heavens there is but one family. It just so happens man that people are different.

既然我们接受我们只是渺小一部分，道法自然，万物皆有道，又互相联系。那么一定我们可以其他”物“领域中学习道，有可能来帮助我们这个”物“这个领域里，一定是这样的。

达芬奇说：

“Principles for the Development of a Complete Mind: Study the science of art. Study the art of science. Develop your senses - especially learn how to see. Realize that everything connects to everything else.”

Sciene and art就是关联着的。引用Da Vinci’s Key to Creativity Came Straight from Nature其中一段黄金分割率：

he celebrated “golden ratio,” a mathematical principle repeated in everything from the arrangement of tree branches to the bones of the human hand, is seen in the master’s work. Known by the Greek letter phi and approximately equal to 1.618, it was first articulated by Leonardo Fibonacci in the year 1202. Nature’s number is all around, even if most of us never notice.

“No self-respecting artist goes around counting tree branches, but Leonardo did,” Atalay said. “He was a scientist doing art. It was always the patterns he was after. Proportions, patterns, the mathematics behind it.”

你需要培养Observation&Sense。观察的能力，开发你的感知能力，正如冥想里当你闭眼时你会集中注意力到你身体的感受上，空气的温度，风吹过的触感，各种声音（切记不要judge，不要评判好听与否，鸟鸣和汽车喇叭声都只是声音），观察你的情绪。或者平时深呼吸，观察周边每一个砖，云的形状，again, don’t judge, just feel,just experience. 不要评判，不要去想，只管去感受，去体验。

Origination&Creativity(创造力）无法被force出来，不能强求出来。

同样人与人情感也是关联的。给自己一点缓冲，跟相近的人发脾气或者对小孩没有耐心即将爆发之前，后退一步，注意到自己的变化，引入写理性思考，思考是自己强加自己的主观意愿忽视别人的了还是真的事实。即使是，有没有更好的处理办法。有些微妙的东西，人与人之间的，正如创造力一样，也不能force出来的，要拖下自己的外套自己的外壳，才能收获心心感应。硬碰硬，短期的宣泄只会是让另外一个人承受痛楚。

很多答案往往并非在自己研究的领域之内（专业之内），可能是在领域之外的广大世界, 因为Mother Nature有所有的答案。我们不能只看见手指或者只知道手指，不知道那个更大光芒的月亮。

理性思维主宰逻辑，但是有些时候创造力来自潜意识。记得很多科学创新的案例，有的来自无意，更有的来自梦境。我这里绝非贬低理性思维的作用，像我说的无为的真意，更好运用理性思维（刀刃上），我们有时候需要给他放一边先。

我有这种感觉，相信大部分人也有这种感觉，当你洗澡的时候，或者比如运动完，（打完篮球，我就很享受酣畅淋漓之后骑自行车时），没有特意思考什么东西，但是有些东西或者想法会自己冒出来，给了一些新的观点或视角。比如写博客的时候，特别容易全身投入，忘乎所以，思绪旁征博引，不需要想，自然而然流露笔尖，不自觉时间过得很快，所以为此我写博客从来不在晚上10点之后写。

感觉像是在潜意识里，它有了理性思维暂时歇息的这个空挡里，它会将知识点等等自己消化，好像理性上面不曾关联的东西（太过专注在局部），在潜意识了找到关联性（万物关联），并且编织成网，最终反馈到理性层面上来。有时候你就是绞尽脑汁就是无法得到答案，往往不经意间往往要得到了答案。

反应到实际，比如编程，我们有时候真的没办法，陷入一个死结之中，这个时候我们可以出去走走散散心，买杯咖啡或者抽个烟（不太建议咯），观察其他的东西，反过头可能会发现也许是一个很蠢的错误，但是思维陷入了这个盲点，看不到森林；所以别人帮你看看，梳理梳理，哪怕是讲解也是一种好的办法。

Andy Hunt写过经典的《The Pragmatic Programmer: From Journeyman to Master》, 大学看的，当时还觉得非常受益。后面他要写了本《Pragmatic Thinking and Learning: Refactor Your Wetware》提到了很多关于认识大脑和自我还有思维修炼的办法 - 其中也有冥想等。

对于设计师和程序员，这种需要高度的脑力活动，我们需要跟健身一样，锻炼我们的大脑，学会正确认识（包括读懂大脑的状态，给出的提示，比如脑子转不动啊或者突然断电），适当的给大脑喘息和缓冲的空间，除了能提高效率包括注意力，说不定有时候也能给一些新的看问题的视角。

打篮球也是这样，禅师Phil Jackson菲尔·杰克逊的书Eleven Rings: The Soul of Success和Sacred Hoops: SPIRITUAL LESSONS OF A HARDWOOD WARRIOR，和他在youtube上和奥普拉的采访How NBA Coach Phil Jackson Taught His Teams Mindfulness，你就知道他真的是讲禅融进了生活，篮球只是他讲他的理念实践一个方面。他像七十年代左翼风暴一样，甲壳虫乔布斯，推崇mindfullness，冥想meditation，推崇铃木俊隆的《禅者的初心》。从他怎么coach乔丹，让他转型做team player团队型球员（我愿意减少得分，给其他团队成员更多机会。ps:结果还是得分王，GOAT就是神），包容bad boys丹尼斯罗德曼的性格，怎么处理我和大我的关系, 建立公牛的两个王朝。包括影响了科比，科比也会每天早上除了凌晨4：30的打铁声，还会做10-15分钟的冥想。Kobe Bryant On Oprah. Meditation Dictates My Day

In fact, I’m set for whatever might come in my way. I have clamness for whatever in my way. Poise.

诗歌历史：作为一名理科生，高中时代我非常高兴去的是历史和诗歌。诗歌或者历史的美妙之处在于他们没有明显的用途，所以喜欢应该说就是纯粹的喜欢。历史和诗歌一样是高度的浓缩，是一个时代一个个人生经历的提炼，当时他因为人生际遇的各有不同掺杂了各种复杂的情绪。读书之时，不觉如何，可能会想设身处地去感受，会去了解作者任务的背景，各种典故，然而还是无法有共鸣之感，毕竟时代不同，无法身临其境。但是往往在自己生活中某个片段，某个瞬间，潜意识了某些诗歌历史画面就会突然冒出来，有一种时空穿梭之感，如此真实，思维火花的碰撞，相当美妙。

苏轼《定风波》

莫听穿林打叶声，何妨吟啸且徐行。

竹仗芒鞋轻胜马，谁怕？一蓑烟雨任平生！

料峭春风吹酒醒，微冷，山头斜阳却相迎。

回首向来萧瑟处，归去，也无风雨也无晴

生活需要平衡，因为平衡之中，我们可以观察万物，观察其道，融会贯通，否则就像一滩死水。

我想我知道要open-minded, 也知道万物是相关联的，知道需要建立自己的知识体系，那么也就知道阅读和观察大自然是多么的重要，也是一生lifelong希望坚持的东西吧。

End

这篇我本意想首尾呼应，但是又感觉想表达的东西太多太杂。可能回应开头是，不忘初心，检视temperament,centered平衡，砥砺前行。如果他们可以活出自己信念，我可以我可以自己摸索实践自己的？

进则全力以赴，退则静定淡泊。

达芬奇：

Be obsessively curious. Observe from details. Never stop learning. Think visually.

乔布斯：

Stay hungry, Stay foolish

《中庸》：

博学之，审问之，慎思之，明辨之，笃行之。

最后：

Strive on, diligently!

Ambassador(API Gateway) + Istio(Service Mesh) 南北+东西

2018-11-30T13:54:32+00:00

熟悉了Kubernetes那些基础概念之后(Service,Pod,Deployment,PV/PVC, StorageClass)等等止呕后，你有了一个cluster集群可以跑起来一些模块的微服务了。接下来就是如何暴露这些微服务能让外部访问？传统的部署中，我们会有Nginx做负载均衡和反向代理，将外部的访问按规则分发给具体的应用服务上。那么在kubernetes里该怎么办？

Ingress

The Ingress is a Kubernetes resource that lets you configure an HTTP load balancer for applications running on Kubernetes, represented by one or more Services. Such a load balancer is necessary to deliver those applications to clients outside of the Kubernetes cluster.

具体可以参考官方的文档说明: Kubernetes Concept: Ingress

那么具体实现的便是ingress controller了。

An ingress can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, and offer name based virtual hosting. An ingress controller is responsible for fulfilling the ingress, usually with a loadbalancer, though it may also configure your edge router or additional frontends to help handle the traffic.

一个简单的Ingress资源示咧：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: myminikube.info
    http:
      paths:
      - path: /
        backend:
          serviceName: frontend
          servicePort: 80
      - path: /auth
        backend:
          serviceName: auth
          servicePort: 7002
      - path: /products
        backend:
          serviceName: products-server
          servicePort: 7002
      - path: /reviews
        backend:
          serviceName: reviews-server
          servicePort: 7002

Ingress-Nginx不足

Ingress-Nginx应该说能大部分满足我们的需求，而且nginx一开始作为高性能的web服务器，慢慢演变支持了更多传统proxy用户案例之外的功能。但是问题是Nginx推出了商业版的Nginx Plus, 收费版本，而且有些功能是Nginx开源版本没有的：比如Nginx Plus就有那个监控的dashboard，不得不nginx log自己摸索。

https://www.nginx.com/products/nginx/#compare-versions

这里我们看到Nginx Plus还是有有一些是Nginx没有的，但是这里我们看到Kubernete Ingress Controller是支持的，但是不支持比如JWT Authentication；而作为一个微服务场景，特别是主要以API为主要实现方式的场景里，认证Authentiction是非常重要的一环，而JWT(JSON Web Token)就是用来保护授权我们API的访问的。

NGINX Plus “extend[s] NGINX into the role of a frontend load balancer and application delivery controller

问题是这个商业化带来很多不确定，你不知道那些功能是会像是JWT一样会被移除开源会挪到企业版本里。

Envoy by Lyft

简单的说envoy就是为微服务量身订造的，更加适合微服务的架构和体系。它的作者在这篇文章Our Move to Envoy详细说明为什么创作它的理由。 Envoy: 7 months later:

In a nutshell, Envoy is a “service mesh” substrate that provides common utilities such as service discovery, load balancing, rate limiting, circuit breaking, stats, logging, tracing, etc. to polyglot (heterogeneous) application architectures.

后面讲到的Ambassador和Istio正是基于Envoy,所以他们的结合应该说是非常丝滑。

Ambassador (JWT) - API Gateway

Ambassador is an open source, Kubernetes-native microservices API gateway built on the Envoy Proxy. Ambassador is built from the ground up to support multiple, independent teams that need to rapidly publish, monitor, and update services for end users. Ambassador can also be used to handle the functions of a Kubernetes ingress controller and load balancer.

Makes it easy to change and add to your Envoy configuration via Kubernetes annotations
Adds the out-of-the-box configuration necessary for production Envoy, e.g., monitoring, health/liveness checks, and more
Extends Envoy with traditional API Gateway functionality such as authentication
Integrates with Istio, for organizations who need a full-blown service mesh

Ambassador是基于Envoy的API Gateway实现。安装还是蛮简单的：部署Ambassador;定义Ambassador服务;最后创建路由

---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind:  Mapping
      name:  httpbin_mapping
      prefix: /httpbin/
      service: httpbin.org:80
      host_rewrite: httpbin.org
spec:
  ports:
  - name: httpbin
    port: 80

这里我们不多言，但是看到好处在于可以在service里直接通过annotation的方式定义路由，这样还是很方便的。

那么我们这里就要讲到JWT，我们就好奇如何在ambassador中引入认证服务。官方文档里有部分提到Basic Authentication但是列子里只是讲到了如何使用基础的basic http auth,，但是问题是我们需要的是JWT的认证方式，所以这个例子并不能直接用，但是好像也没有其他JWT的现存的可以直接用。

好在我们看到其中的一个章节：The External Authentication Service

首先我们写一个JWT的decode的middleware:

var jwt = require('jsonwebtoken');
var config = require('../config.js');

module.exports = function(req, res, next) {
    /*
     * Check if authorization header is set
     */	    
     var token = ""
     if(req.cookies && req.cookies.id_token){
        token = req.cookies.id_token
     }else if (req.headers.authorization && req.headers.authorization.split(' ')[0] === 'Bearer') {
        token = req.headers.authorization.split(' ')[1];
     } else if (req.query && req.query.token) {
        token = req.query.token;
     }
    if(token) {
        try {
            /*
             * Try to decode & verify the JWT token
             * The token contains user's id ( it can contain more informations )
             * and this is saved in req.user object

             { name: 'Admin',
              email: 'admin@qq.com',
              id: 1,
              iat: 1543476317,
              exp: 1543562717 }

             */

             const result = jwt.verify(token, config.JWT_SECRET);
             console.log("result", result)
             res.set('x-admin-id', result.id);
             res.set('x-admin-email', result.email);
            req.user = result;
        } catch(err) {
            /*
             * If the authorization header is corrupted, it throws exception
             * So return 401 status code with JSON error message
             */
            return res.status(401).json({
                error: {
                    msg: 'Failed to authenticate token!'
                }
            });
        }
    } else {
        /*
         * If there is no autorization header, return 401 status code with JSON
         * error message
         */
        return res.status(401).json({
            error: {
                msg: 'No token!'
            }
        });
    }
    next();
    return;
};

然后post login的代码里我们从body里提出username/password然后校验:

app.post('/extauth/login', (req, res, next) => {
  const {username, password} = req.body;
  const defaultEmail = "admin@qq.com";
  const defaultName = "Admin";
  const defaultPassword = "123";
  if (username !== defaultEmail || password !== defaultPassword) {
      console.log("========= DONT MATCH")
      res.status(400).json({ message: 'Username or password is incorrect' })
      return;
  }
  const data = {name: defaultName, email: defaultEmail, id: 1};
  const expiresIn = 60 * 60 * 24; // 24 hours
  const token = jwt.sign(data, jwtSecret, {expiresIn});
  res.cookie('id_token', token, {maxAge: expiresIn * 1000, httpOnly: true});
  return res.json({admin: data, id_token: token});  
});
	
app.all('/extauth/nodebackend/', authMiddlware, function (req, res, next) {
  console.log("nodebackend - req user: ", req.user);
  res.send('OK (authenticated)')
  //next();
})

app.all('*', function (req, res, next) {
  console.log(`Allowing request to ${req.path}`)
  res.send('OK (not /qotm/quote)')
  //next();
})

对于/nodebackend的路由，我们让其被保护，其他的通过。但是跑起来去登录发现， req.body输出是空，就是说username/password始终是null，发生了什么事情？

然后我们看ambassador文档：

For every incoming request, the HTTP method and headers are forwarded to the auth service. Only two changes are made:

* The Content-Length header is overwritten with 0.
* The body is removed.

Envoy (through patches we’ve contributed) allows the attachment of arbitrary metadata to service instances, and the definition of routing rules based on that metadata.

所以Ambassador会将http method和headers来传给Auth service, 记住所有方法都会先进入auth service再转发到实际的serivce或者被踢回。

这就是为什么了body是空的？所以我们这里必须将username/password作为http basic auth的方式传递进来。至于怎么做？你可以参考： ambassador-auth-service

Allowing the Request to Continue (HTTP status code 200) To tell Ambassador that the request should be allowed, the external auth service must return an HTTP status of 200. Note well that only 200 indicates success; other 2yz status codes will prevent the request from continuing, as below.

Preventing the Request from Continuing (any HTTP status code other than 200)

所以我们这里 return res.json({admin: data, id_token: token}); 会导致404错误，因为/login在其他服务上没有实现。。。但是我们要不能传递500，401吧。

所以这里我们特殊处理下返回…201.

res.status(201).send({admin: data, id_token: token});

所以当其他服务想要获取当前认证的用户时，req.user将不会有作用，你需要从header上拿。这里你需要明确在allowed_headers加上x-admin-id和x-admin-email或者啥的。

---
apiVersion: v1
kind: Service
metadata:
  name: example-auth
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind:  AuthService
      name:  authentication
      auth_service: "example-auth:7002"
      path_prefix: "/extauth"
      allowed_headers:
      - "x-qotm-session"
      - "x-admin-id"
      - "x-admin-email"
spec:
  type: ClusterIP
  selector:
    app: example-auth
  ports:
  - port: 7002
    name: http-example-auth
    targetPort: http-api

Istio: Service Mesh

如果说ambassador解决是纵向的通信分发，那么Service Mesh(istio)就是解决横向微服务之间的交通通信。

Istio gives you:

Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic. Fine-grained control of traffic behavior with rich routing rules, retries, failovers, and fault injection.

A pluggable policy layer and configuration API supporting access controls, rate limits and quotas.

Automatic metrics, logs, and traces for all traffic within a cluster, including cluster ingress and egress.

Secure service-to-service authentication with strong identity assertions between services in a cluster.

这里的Proxy就是基于Envoy的，基本上作为sidecar拦截所宿主的service的网络通信并且向pilot中央报告，然后根据规则等等(mixer)做出服务查找通信等等。

Ray Tsang: Making Microservices Micro with Istio and Kubernetes演示的非常好，可以看看。

Ambassador and Istio: Edge Proxy and Service Mesh

Ambassador is a Kubernetes-native API gateway for microservices. Ambassador is deployed at the edge of your network, and routes incoming traffic to your internal services (aka “north-south” traffic). Istio is a service mesh for microservices, and is designed to add application-level Layer (L7) observability, routing, and resilience to service-to-service traffic (aka “east-west” traffic). Both Istio and Ambassador are built using Envoy.

所以看起来Ambassador和Istio合并看来是非常不错的组合。可以参考官方的博客： https://www.getambassador.io/user-guide/with-istio/ 这里并没有啥特别的，这里实战的话还是用Helm比较方便干净点。

参考

Envoy vs NGINX vs HAProxy: Why the open source Ambassador API Gateway chose Envoy

Part 3: Deploying Envoy as an API Gateway for Microservices

API Gateway vs Service Mesh

Managing service mesh on Kubernetes with Istio

Kubernetes in Action

2018-10-21T12:29:46+00:00

一年前还写了一篇docker入门的博客，回头看看，其实一直也没机会花太多心思去搞，因为目前这个jenkins + github + pm2部署虽然说不上特别体面，但是对于刚刚开始的小项目或者说大部分项目目前的流量都是非常小的，本着这个lean原则，基本上都够用。感觉用上docker或者kubernetes有点牛刀小试，一方面镜像的推送拉取，虽然通过镜像加速，仍然不免有点费时费事；另外一方面没有大型应用的场景，为了技术而技术貌似有点多余。基本上因为环境不一致导致出现BUG的几率其实是相当少见的。当然因为最近有这么一个多租户的问题，需要做一些技术研究，同时抱着学习的态度，开始了解了下kubernetes. 在这个搭建和入门的过程中，我更加觉得系统架构的演化是需要配合实际业务现实场景的要求的。这中间主要是看了kubernetes官方的一些教程，感觉还是有点稀里糊涂；后面看到了一本kubernetes in action，由浅入深讲的真是非常不错。

Hello World

学习一个新技术了解一个新东西，好比跟学车学驾驶一辆车辆类似的：直接上来讲道理，估计没人有兴趣了解；但是如果你让他先倒腾两下，转转方向盘，踩踩油门，挂挡，在空白的道路上来这么两下，有了这个亲身体会，后面你跟他讲这个原理，那么他能联系起来理解也更到位。

Minikube 可以让你最快的在本地跑起来在虚拟机内部的一个单节点的Kubernetes集群/Cluster。跑了一遍之后，你大概就知道这玩意还是蛮重的,还有很多的概念名词，物理上的服务器网络，还有核心概念(Service, Pod, Node, ReplicationController, Replicaset, Deployment…)等等，相当的头大。

当然跑个本地单机能让你基本了解，但是很多东西没办法真的模拟真实的的多节点的情况。比对了下，大致一般两种，一种是自建的，还有一种是用云服务商提供的。可以参考阿里云 Kubernetes vs. 自建 Kubernetes

亚马逊AWS有EKS，Google有GKE，听起来是挺不错的，但是问题是直接用云的了，一方面比较贵，特别是阿里云，master node必须3个，加上两个node,随便一下一个月就一万多快，坑爹；另外一方面自建能够帮助更好的理解kubernetes（虽然碰到很多坑 NAS, EIP, ELB, ALB, EBS) 算了下，假设咱模拟的话至少一个master节点+两个node节点，这才是一个体面的集群，也符合咱体面程序员的最低要求。大致看了看，发现kops仓库的帖子的讨论里Estimated costs for smallest cluster on AWS & GCE?谈到了看起来还蛮便宜啊，一个月$150美元，包括3台t3.medium的机子加上128G EBS存储, 一个ELB(负载均衡)还有10MB的s3存储（需要存储集群的配置和元数据）。感觉可以搞一把试试。

在AWS上搭建kubernetes

首先搭建Kuberetes集群，虽然官方提供了kubeadmin, 但是kops显然更加方便强大。

可以参考Installing Kubernetes on AWS with kops 或者这篇文章Setting Up a Kubernetes Cluster on AWS in 5 Minutes

命令加起来并不多，但是有一些要注意的地方。一个是地区region要保持一致，比如我这里选择了us-west-2（Oregon); 另外一个是

kops create cluster --node-count=2 --node-size=t2.micro --master-size=t2.micro --zones=us-west-2a

这里–node-size和–master-size选择是明确的指定了t2.micro 也就是免费级别AWS Free Tier, 理论上说免费一个节点一个月，因为我们创建了3个，所以可能咱们只能玩1/3个月，后面就需要收费了，但是相比之前$150那个t3.medium还是便宜不少，虽然只有一个CPU，后面试了试，也没什么问题（GPU和内存都没有爆，即使创建了很多Pod),但是什么ELB就得收费了尽管便宜, s3有5G足够够用。

跑完之后，运行 kops validate cluster, 应该看到上面的输出，然后去AWS的console里验证一下有没有三个ec2实例运行。

看起来不错，现在可以装一个web ui的kubernete dashboard来看看。

kubectl

搭建了好了集群，你现在可以通过kubectl命令来远程操作集群了。 Overview of kubectl 提到了它的各种命令和详细用途，比如好用的就是这个缩写。但是经常打这个kubectl get po -o wide 或者其他的命令，有时候过于繁琐，虽然缩写可以减少部分的打字。

We’re using the abbreviation rc instead of replicationcontroller.
Most resource types have an abbreviation like this so you don’t have to type
the full name (for example, po for pods, svc for services, and so on).

这时候可以考虑用到zsh的补全提示zsh-kubectl-prompt和插件Kubectl plugin来简化打字。

Kubernetes in Action

这本书由浅到深的讲解了为什么使用容器技术，以及kubernetes的基础知识和概率以及最后深入探究内部实现。

正如之前你看到的那么多kuberenetes的核心概念，还包括物理上的各种技术网络存储服务器通讯等等，官方文档还是无法一步一步带你入门深入串起来来更好的理解，而不是被这些术语给淹没掉。而kubernetes的核心概念：一个是imperative,一个是abstraction/encapsulation. 封装底层实现的细节，比如deployment -> replicasets -> pods 等等从而暴露出最简单明了而且风格统一的操作命令，这个封装还包括物理上的。

PDF里我做了很多笔记，但是太多了，无法贴上来，总而言之这本书应该非常适合入门。动手完helloworld自后，在看这本书，感觉很多东西都豁然开朗。这本书你应该只需要读前两部分，大概是接近300页英文左右，后面一部分讲的是内核和实现，后面有经验了可以反过头来阅读。

花费

大家可以看看3台EC2其实只跑了188个小时，离750小时还是差蛮多的；EBS是我测试PersistentVolume一部分；这里面有一个ELB（Elastic Load Balancing)的计费有点意思。

其实这个ELB还是蛮贵的，很多现实的里面很大部分费用因为过多的ELB实例产生的。为什么会有这么多? 它怎么来的了？

The big downside is that each service you expose with a LoadBalancer will get its own IP address, and you have to pay for a LoadBalancer per exposed service, which can get expensive!

也就是当你expose暴露服务给外部访问时，选的是load balancer类型的话，亚马逊平台的负载均衡会分配一个实例给它用来给外部访问，所以你可以想象。

怎么办了？这里要提到上面截图里有一个Ingress.

相当于ELB和Service中间加了一层. 这样减少ELB的使用，甚至一个就可以了。

AWS Cost Savings by Utilizing Kubernetes Ingress with Classic ELB

Mac环境下使用Homebrew安装Docker

2017-08-21T16:51:52+00:00

如果我没记错，几年前还在使用Puppet和Chef来快速provisioning AWS上的机器，虽然说比手动登一个一个SSH登上去手工操作是方便了些，但是还得写各种Recipe, 也是一件头疼的事情。现在有了Docker, DevOps甚至结合slack的ChatOps, 使得持续部署变得看起来是件容易的事情了。

Docker

Docker是一下在MAC下面安装Docker，过去你可以使用Docker Tools加上笨拙的虚拟机Virtualbox, 但是还是比较麻烦的. 当然你可以去Docker.com去下载.dmg然后一键安装。问题在OSX上面，宿主机器和容器之间文件共享会变得非常慢。当然如果你只是小试身手，这个方式也没问题。这里我们将着侧重在使用Homebrew + xhyve/hyperkit(native macOS hypervisor.framework)来安装设置好docker.

安装

确保您安装好Homebrew,然后去terminal中打开

brew update
brew install docker docker-compose docker-machine xhyve docker-machine-driver-xhyve

这里需要解释下xhyve:

Why xhyve ?

So one of the painful points of using docker on OS X is that you need to run a virtualbox VM, which often suffers from performance issues. With xhyve, a OS X virtualization system, and docker-machine-xhyve you can now have docker use the native OS X hypervisor to run containers.

No more dealing with virtualbox shenanigans!  

简单说轻量级的OSX虚拟化解决方案，让你摆脱恶心的VirtualBox设置和更新。

接下来在terminal中，按照提示运行root权限命令：

sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve  

就是这么简单。接下来就用docker-machine创建一个machine吧！

阿里云Docker源加速

DockerHub确实厉害，各种各样的镜像你都能找到，唯一问题是就是镜像，你懂得，基本上都上百兆上G，国内访问这速度真的感人。

幸好就像rubygem有淘宝镜像一样，阿里云也提供Docker加速器。进入https://cr.console.aliyun.com，注册好账号，找到Docker Hub镜像站点，在上面找到你专属加速器地址:

https://xxx.mirror.aliyuncs.com

记下来。

创建machine

回到termial, 输入:

# Uninstall Docker Toolbox:
sh -c "$(curl -fsSl https://raw.githubusercontent.com/docker/toolbox/master/osx/uninstall.sh)"
#And remove existing caches
sudo rm -rf ~/.docker

主要是清除之前Docker安装的信息。

接下里输入：

docker-machine create default --driver xhyve --xhyve-experimental-nfs-share --engine-registry-mirror=https://xxx.mirror.aliyuncs.com

注意我们在后面添加了--engine-registry-mirror=https://xxx.mirror.aliyuncs.com赋予了阿里云提供的专属加速器。

Hello World

创建machine顺利的话，在terminal输入:

docker pull hello-world
docker run hello-world

如果看到以下输出就表示docker安装并设置成功:

⇒  docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://cloud.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/engine/userguide/  

Docker images

你可以抓取一些常用的镜像，比如docker pull node:

⇒  docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
node                  latest              6f6ffe2a1302        2 days ago          669MB
hello-world         latest              1815c82652c0        2 months ago        1.84kB

引用

用树莓派打造流光溢彩 - Build Ambilight with Raspberry Pi (二)

2016-01-30T14:21:30+00:00

在我们迫不及待的链接LED灯带使其发光之前，我们先了解下LED灯带的结构和参数，因为如果一旦接线错误，有些后果是无法挽回的，可能整个灯带将会被烧掉。

LED灯带结构

接头部分有一个接头从左到右有黑，绿，红，蓝色的线。另外一段有也有对应四个连接引脚5V,CK,SI和GND. 特别需要注意灯颗粒上面的箭头，这个是因为灯带的发光方向是有要求的，必须是单方向，从输入到输出，特别是后面布到电视机后面需要将灯带剪开，然后焊接到一起，这个时候必须保证箭头的方向是一致的。

LED灯带布线电视机

到了动手的环节了，我们会将LED灯带布线到电视机后面。LED灯带背面有双面胶，将其拨开就可以贴在电视机上面了。我们先从电视机的右边（站在电视机前面面对电视机来看的右边）开始，分成3段，右边->上边->左边，逆时针方向将其布好到电视机上。这里需要保证每一段上面的LED灯带的箭头是一致的。

焊接连接头的时候，记住5V对5V,CK对CK, SI对SI，GND对GND。这是个技术活，记得别紧张慢慢来，虽然焊接圆点不打，其实焊接起来还是比较简单的。

LED灯带跟树莓派接线

接下来我们需要给LED灯带连线，我们首先给灯带供电，我们将拿杜邦线将灯带的5V黑色端连接到DC电源母头的正(+)端，GND蓝色链接到DC电源母头的负(-)端。将LED灯带上的CK（时钟信号）连接到树莓派上的SCLK内行第二个引脚上，SI（数据信号）链接到树莓派上内行第四个MOSI引脚。

这里先上一张连线图，在Phillip Burgess的基础上，我修改了一下：

</br>

这里树莓派上的GND和5V引脚不是一定需要链接到电源上，大可不必纠结，我只是图方便省事。

连接其他组件

最后我们把其他组件按顺序都连接起来。

软件端准备Hyperion

目前为止，硬件端基本已经差不多，我们接下来需要准备软件端。

操作系统 - OSMC.

对于初学者而言,OSMC比较简单直观也有可视化的桌面，上手比较快。我们需要去下载页面，选取合适的镜像和平台，下载之后将SD卡插上电脑直接烧录进去即可。烧录好之后查到树莓派的卡槽中并启动，记得插上网线，然后获取IP地址。第一次启动需要你用键盘配置下语言，时区等。

记下IP地址:192.168.1.3

LED灯带控件软件 - Hyperion.

我们需要Hyperion来读取视频截图，然后转换信号并映射到LED灯带上。Hyperion有很多的有点，比如低CPU使用率，后台运行，轻量级，速度快等等，总之是为树莓派量身打造。更重要的是它提供了一个图形化的界面来配置灯带映射的参数等等。

第一步我们需要SSH登录到树莓派:

    ssh osmc@192.168.1.3

OSMC默认的账号是:osmc/osmc。输入密码osmc即可登录。

第二部安装Hyerion，可以根据官方文档Installation on OSMC RC3:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install git libqtcore4 libqtgui4 libqt4-network libusb-1.0-0 libprotobuf9 ca-certificates
wget -N https://raw.github.com/tvdzwan/hyperion/master/bin/install_hyperion.sh
sudo sh ./install_hyperion.sh
sudo vi /boot/config.txt（然后末尾添加此行：dtparam=spi=on)

Hyperion视频采集卡调试

安装好后，可以快速调试一下看看是否工作：

    hyperion-remote --priority 50 --color red --duration 5000

看看是否都亮红色，比如下图我这里是贴上电视机前先测试的：

接下里我们测试下视频采集卡模块是否正常工作。

先检测是否视频采集卡被识别。

    apt-get install lsusb

运行之后会有如下输出：

osmc@osmc:~$ lsusb
Bus 001 Device 004: ID 05e1:0408 Syntek Semiconductor Co., Ltd STK1160 Video Capture Device
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. LAN9500 Ethernet 10/100 Adapter / SMSC9512/9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
osmc@osmc:~$

Ltd STK1160 Video Capture Device 那一行就表示被识别。

调试Hyerion V412 Grabber模块。

根据官方文档V412 Grabber, 我们先通过命令调试下部分参数来确保能得到正确的视频截图。

    sudo hyperion-v4l2 --width -1 --height -1 --size-decimator 1 --frame-decimator 1 --screenshot

会得到一个screenshot文件，是左边这样的。

您看到有黑边，这里我们需要配置crop margin，截取黑边部分，

sudo hyperion-v4l2 --width -1 --height -1 --crop-top 10 --crop-left 30 --crop-bottom 10 --crop-right 40 --size-decimator 1 --frame-decimator 1 --screenshot

就得到右边的截图。

Hyperion配置LED灯带映射

到了配置Hyperion的时候了，你可以通过Hyerion的图形化工具来配置。工具的文档Hyerion Configuration有详细的工具使用说明，先下载工具HyperCon_ssh.jar. 下载到本地之后运行java -jar HyperCon_ssh.jar：

type: 选择WS2081 RGB byte order: RGB direction: 方向选取逆时针 Led in top/bottom coners: 因为我们在顶角90度的地方并没有灯叠加，所以都选false Horizontal: 水平灯一共有32个 Vertical: 左右边有19个 Bottom Gap: 因为底边没有灯，所以设置为顶边的灯数来去掉底边 1st LED offset: 设置此参数来确保第一个标号0从灯带的输入端开始，也就是右边地下第一个。

后面的Horizontal depth等等可以保持默认。

接下来设置采集卡参数，禁止Frame Grabber, 同时开起GrabberV4L2, 将下面的参数按照调试的时候填上。最后点击下面的Create Hyerion Configuration,导出hyperion.config.json配置文件,并SCP上传到树莓派上。

scp hyperion.config.json osmc@192.168.1.3:~
登陆到树莓派
cp hyperion.config.json /etc/
sudo /etc/init.d/hyperion stop

怎么调试了？可以运行如下命令来查看输出：

sudo hyperiond /etc/hyperion.config.json

会有如下输出：

Application build time: Jan 30 2016 16:59:41 QCoreApplication initialised Selected configuration file: /etc/hyperion.config.json ColorTransform ‘default’ => [0; 203] Device configuration: { “colorOrder” : “rgb”, “name” : “MyPi”, “output” : “/dev/spidev0.0”, “rate” : 250000, “type” : “WS2801” } Black border threshold set to 0.1 (26) Creating linear smoothing Created linear-smoothing(interval_ms=25;settlingTime_ms=200;updateDelay=0 Effect loaded: Knight rider Effect loaded: Blue mood blobs Effect loaded: Cold mood blobs Effect loaded: Full color mood blobs Effect loaded: Green mood blobs Effect loaded: Red mood blobs Effect loaded: Warm mood blobs Effect loaded: Rainbow mood Effect loaded: Rainbow swirl fast Effect loaded: Rainbow swirl Effect loaded: Snake Effect loaded: Strobe blue Effect loaded: Strobe Raspbmc Effect loaded: Strobe white Initializing Python interpreter Hyperion created and initialised run effect Rainbow swirl fast on channel 0 Boot sequence(Rainbow swirl fast) created and started V4L2 width=720 height=480 V4L2 pixel format=UYVY V4L2 grabber signal threshold set to: {25,25,25} V4L2 grabber started V4l2 grabber created and started …

至此，就表示大功告成。

最后运行sudo /etc/init.d/hyperion start来启动Hyerion. 尽情享受流光溢彩Ambilight吧~

引用

用树莓派打造流光溢彩 - Build Ambilight with Raspberry Pi (一)

2016-01-27T14:21:30+00:00

最近翻出了很久之前买的树莓派(2012 第一代的)，一看搁那也没有什么用，加上最近整了个PS4，于是想如何把树莓派利用起来然后跟PS4搭配起来整点啥。网上搜了搜，淘宝上买了些配件，自己回来琢磨了下决定搞一个流光溢彩Ambilight.最后弄出来的效果还是不错的，下面是两张调试后之后的效果图(还有些颜色需要调调，但是大概差不多):

这系列博客主要记录下这个过程，毕竟国外有博客介绍Ambilight，但是配件啥还是得淘宝上买，然后自己倒腾的过程也碰到了一点坑，总结出来也是抛砖引玉。

原理

以本码农多年挖坑填坑的惨痛经历来看，开始的理论文档虽然比较枯燥点，但是对后面实践会有很大的帮助。先上一张原理流程图（引用自http://bite-in.com/?p=9,我在这上面注释了下）：

Ambilight原理其实很简单，首先您需要一个视频源，比如我这里是PS4，是IPTV，是电脑等等，这里必须是HDMI的高清信号源。接下来需要个HDMI的分配器1进2出，进的一端连接到信号源，出的两个端口，一个连接到电视，另外一个给树莓派来处理。但是你不能直接将HDMI信号给树莓派使用，暂时目前没有办法抓取HDMI信号，所以得退回使用传统的AV模拟信号，所以这里需要个将HDMI转AV的转换器.这里拿到视频Video信号之后，需要一个USB视频帧抓取器，也就是USB接口的视频采集卡，将视频信号转换为USB输出，输入给树莓派。树莓派拿到视频帧信息之后，可以根据预先的配置信息，这里比如有多少个LED灯，方向，色值，颜色转换，频率等等，输出到LED灯带，进而使其按照我们的设想来发光。

配置LED灯带的发光序列和映射是通过树莓派基于OSMC系统上的Hyerion来操作，我会在接下来的部分中仔细描述。

配件清单

分析了原理之后，就需要准备和采购配件了，这里我会列举出组件和淘宝的链接和价格，方便各位参考。

Raspberry Pi(树莓派)：树莓派是这个实验中最核心的部分，作为控制中心。我的树莓派是2012年时在官方Elements 14买的，当时的配置是256M内存，各项配置远远没有现在高，但是貌似价格也没有太大幅度的涨价。因为树莓派的操作系统是存储在从外部的闪存卡中。所以记住同时买一个8G的闪存卡还有一个读卡器。（价格：250左右）

LED灯带:这个是比较关键的零件，我使用的是5V的WS2801型号，这个型号是比较基本和简单的。我量过我的电视上+左+右（因为一般来说底边是不需要发光的），所以买了3米。淘宝链接,选取白色裸管，价格：39.9 * 3 = 120。

HDMI分配器1进2出：需要将一个高清信号渠道分解到两个。淘宝链接价格：58

USB视频采集卡: 这里我们使用基于STK1160的EasyCap DC60型号。淘宝链接价格： 24

HDMI转AV线转换器：高清信号转模拟信号，就需要顶多带几米的LED灯带，也不需要高清信号那么奢侈，模拟信号刚好。淘宝链接价格：68

DC 5V/8A电源一个：记住这个电压非常关键，本实验中所有耗电组件额定电压是5V。如果你只是要带LED灯带，这个足够了。但是如果你想像我一样，希望用一个电源供给所有的组件，比如LED灯带，树莓派，HDMI分配器，以及HDMI转AV的转换器，那其实我发现买的功率不是很够带动所有的组件，买一个5V/12A 或者 5V/20A的会比较稳妥，我后面会描述如何计算功率来选取合适的电源。淘宝链接价格： 30.

HDMI高清线若干条。淘宝链接
AV公对公直通头：需要将视频采集卡和HDMI转AV线转换器连接起来。淘宝链接价格: 4

杜邦线公对母公对公：杜邦线可以帮助连接树莓派和LED灯带，以及LED灯带分解和焊接。淘宝链接

DC电源插座5.5-2.1mm 母头: 这里我们需要将它和LED灯带的供电接头连接起来实现对灯带的供电。淘宝链接

DC母头转Micro USB: 实现对树莓派供电。（如果你不是集中一起供电，这个不是必选的）淘宝链接

DC母头转Mini USB: 实现对HDMI2AV转换器供电。（如果你不是集中一起供电，这个不是必选的）淘宝链接

DC电源分线一拖四：将电源转4路，供给树莓派，LED灯带，HDMI2AV, HDMI分配器。（如果你不是集中一起供电，这个不是必选的）淘宝链接

还需要一些焊接LED灯带需要的洛铁，焊锡和焊锡膏，我买的就是住的地方下面的五金店20块的那种，用起来也挺顺手的，丢了也不觉得多肉疼。一个USB键盘（20块）来实现基本的树莓派系统设置，几米长的一根网线给树莓派网络来实现从电脑上远程SSH控制操作。我买了一个USB无线网卡淘宝链接，来配置一下无线网络，这样不用拖着很长的网线，不过这篇博客不会描述这一部分，使用有线网络比较简单。

LED灯带功率计算

这里需要计算LED灯带消耗的功率是多少。从WS2801的技术规格来看，它的工作电压是5V，至于每一颗灯粒大概消耗0.3W，所有有了这些我们就只需要知道安装到我的电视43寸需要多长的灯带即可。大概比划了下，记住这里我不需要电视机底部的LED灯带（因为我的电视是放在桌子上的，并不是挂在墙上，底部灯光效果会有影响，所以去掉底部的灯带），需要灯带的长度为2.6m. 每米LED灯带有32颗灯点，所以这里我们就知道我们需要多少瓦特的功率了。

2.6米的灯带 * 32颗每米 * 0.3W = 24.96 W 也就是 24.96W/5V = ~5A的电流 

但是建议加上10%的损耗，功率大点总是好的，不然小了还亮不起来。

如果你是给个组件单独供电，那么给灯带选取的电流建议选取6A(5A加上损耗)，也就是5V 6A的电源。

如果你是给所有的组件供电，对的很幸运，所有得组件的额定电压都是5V。那么我们还的加上树莓派的1A，HDMI转AV的1A，HDMI分配器的1A（这些请查阅技术规格，淘宝页面上有的），最后需要8A在加上10%的损耗，建议买5V 10A的电源。

接下来

这里我们基本上理清楚了我们需要买那么组件和基本的原理，那么下一篇就到了动手的环节了，我们需要测试LED灯并布线到电视机上。

android permission handling on marshmallow and preM

2015-12-21T14:24:35+00:00

Before android marshmallow, permission prompt, e.g. making a call, reading contacts, retrieving user location, happens in Google Play when you press install button.

On marshmallow, story is little bit different - it kinda looks like what iOS does. Official android developer website has a dedicated section for permission handling: Working with System Permissions.

When I try to get this new stuff work in practice, two things worth some elaborations: Never ask again and pre marshmallow handling.

`Never ask again` checked

The new permission workflow works like following. Suppose, there is a button that you could press to make call.

private static final int PERMISSIONS_REQUEST_CALL_PHONE = 201;
private String mManifestPersmission;
private int mRequestCode;

mManifestPersmission = Manifest.permission.CALL_PHONE;
mRequestCode = PERMISSIONS_REQUEST_CALL_PHONE;

int permerssion = ActivityCompat.checkSelfPermission(mActivity, mManifestPersmission);                     boolean should = ActivityCompat.shouldShowRequestPermissionRationale(mActivity, mManifestPersmission);            
if (permerssion != PackageManager.PERMISSION_GRANTED) {
    requestPermission();
}

private void requestPermission() {
    ActivityCompat.requestPermissions(mActivity, new String[]{mManifestPersmission}, mRequestCode);
}

And when you press the call btn first time, it would prompt:

If you choose ALLOW, it would just go and call like pre marshmallow; If you choose DENY, it means you have denied permission for user to access, then better we’d show an alert to guide user what’s going like following screenshot:

The idea here is, so user has denied permission, we need provide some explanation(i.e. rationale) to tell user why we need it. Also we provide two actions: Retry and I’m Sure. Click Retry, it would prompt permission asking alert again; Click I’m sure, then it just dismiss silently because user has explicitly known what he is doing.

So if you clicked Retry or you press call button second time, the permission requesting window will have an option: Never ask again.

The tricky part here what happens if user has denied with Never ask again checked? It turns out in onRequestPermissionsResult, you could query shouldShowRequestPermissionRationale to tell whether user has denied with Never ask again option.

The code in onRequestPermissionsResult:

public void onRequestPermissionsResult(int requestCode, String permissions[], int[] grantResults) {
    if(requestCode == mRequestCode){
        Logger.t(mManifestPersmission);
        boolean hasSth = grantResults.length > 0;
        if(hasSth){
            if(grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                //user accepted , make call
                Logger.d("Permission granted");
                if(this.mAffirmativeCallback != null){
                    this.mAffirmativeCallback.onPermissionConfirmed();
                }
            } else if(grantResults[0] == PackageManager.PERMISSION_DENIED) {
                boolean should = ActivityCompat.shouldShowRequestPermissionRationale(mActivity, mManifestPersmission);
                if(should){
                    //user denied without Never ask again, just show rationale explanation
                    AlertDialog.Builder builder = new AlertDialog.Builder(mActivity, R.style.AppCompatAlertDialogStyle);
                    builder.setTitle("Permission Denied");
                    builder.setMessage("Without this permission the app is unable to make call.Are you sure you want to deny this permission?");
                    builder.setPositiveButton("I'M SURE", new DialogInterface.OnClickListener() {
                        @Override
                        public void onClick(DialogInterface dialog, int which) {
                            dialog.dismiss();
                        }
                    });
                    builder.setNegativeButton("RE-TRY", new DialogInterface.OnClickListener() {
                        @Override
                        public void onClick(DialogInterface dialog, int which) {
                            dialog.dismiss();
                            requestPermission();
                        }
                    });
                    builder.show();

                }else{
                    //user has denied with `Never Ask Again`, go to settings
                    promptSettings();
                }
            }
        }
    }
}

private void promptSettings() {
    AlertDialog.Builder builder = new AlertDialog.Builder(mActivity, R.style.AppCompatAlertDialogStyle);
    builder.setTitle(mDeniedNeverAskTitle);
    builder.setMessage(mDeniedNeverAskMsg);
    builder.setPositiveButton("go to Settings", new DialogInterface.OnClickListener() {
        @Override
        public void onClick(DialogInterface dialog, int which) {
            dialog.dismiss();
            goToSettings();
        }
    });
    builder.setNegativeButton("Cancel", null);
    builder.show();
}

private void goToSettings() {
    Intent myAppSettings = new Intent(Settings.ACTION_APPLICATION_DETAILS_SETTINGS, Uri.parse("package:" + mActivity.getPackageName()));
    myAppSettings.addCategory(Intent.CATEGORY_DEFAULT);
    myAppSettings.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
    mActivity.startActivity(myAppSettings);
}

So if shouldShowRequestPermissionRationale returns false, we will display an alert for go to app settings to allow user manually toggle permissions.

Last point, when request permission, we need a way to distinguish two case when shouldShowRequestPermissionRationale returns false. According to android documentation:shouldShowRequestPermissionRationale

To help find the situations where you need to provide extra explanation, the system provides the Activity.shouldShowRequestPermissionRationale(String) method. This method returns true if the app has requested this permission previously and the user denied the request. That indicates that you should probably explain to the user why you need the permission.

If the user turned down the permission request in the past and chose the Don't ask again option in the permission request system dialog, this method returns false. The method also returns false if the device policy prohibits the app from having that permission.

when shouldShowRequestPermissionRationale return false, it could be either user request permission first time or user has denied with Never ask again before. Hence we need modify request code a little bit:

private static final int PERMISSIONS_REQUEST_CALL_PHONE = 201;
...
boolean should = ActivityCompat.shouldShowRequestPermissionRationale(mActivity, mManifestPersmission);            
if (permerssion != PackageManager.PERMISSION_GRANTED) {
    if (should) {
        // should show some explanation alert, but here now, just prompt ask again                    
        requestPermission();
    } else {
        //TWO CASE:
        //1. first time - system up - //request window
        if(!PrefUtils.hasLocationPermissionBeenRequested(mActivity)){
            PrefUtils.markLocationPermissionBeenRequested(mActivity, true);                     requestPermission();
        }else{
            //2. second time - user denied with never ask - go to settings                     promptSettings();
        }
    }
    return;
}

That’s all.

Pre marshmallow and code reuse

To make it work with pre marshmallow, we could encapsulate those logic in a helper. So in helper, we could specify a general callback for affirmative actions. On pre marshmallow, you could just call that callback; on marshmallow, do permission flow:

public class PermissionHelper {

    public interface PermissionAffirmativeCallback
    {

        public void onPermissionConfirmed();
    }
    
    private PermissionAffirmativeCallback mAffirmativeCallback;

    public static PermissionHelper permissionHelper(PermissionType type,
                                                        Activity activity,
                                                        PermissionAffirmativeCallback callback){
            return new PermissionHelper(type, activity, callback);
        }

    public PermissionHelper(PermissionType type, Activity activity, PermissionAffirmativeCallback callback) {

        if(type == PermissionType.LOCATION){
            mManifestPersmission = Manifest.permission.ACCESS_FINE_LOCATION;
            mRequestCode = PERMISSIONS_REQUEST_LOCATION;
            mDeniedMsg = "Without this permission the app is unable to find your location.Are you sure you want to deny this permission?";
            mDeniedNeverAskTitle = "Unable to locate your position";
            mDeniedNeverAskMsg = "You have denied the permission for location access. Please go to app settings and allow permission";
        }else if(type == PermissionType.CALL){
            mManifestPersmission = Manifest.permission.CALL_PHONE;
            mRequestCode = PERMISSIONS_REQUEST_CALL_PHONE;
            mDeniedMsg = "Without this permission the app is unable to make call.Are you sure you want to deny this permission?";
            mDeniedNeverAskTitle = "Unable to make call";
            mDeniedNeverAskMsg = "You have denied the permission for calling.. Please go to app settings and allow permission";
        }
        this.mActivity = activity;
        this.mAffirmativeCallback = callback;
        checkPermission();
    }
    

    private void checkPermission() {
       if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
           int permerssion = ActivityCompat.checkSelfPermission(mActivity, mManifestPersmission);               
           boolean should = ActivityCompat.shouldShowRequestPermissionRationale(mActivity, mManifestPersmission);               
           if (permerssion != PackageManager.PERMISSION_GRANTED) {
               //...blablabla
               return;
           }
       }

       if(this.mAffirmativeCallback != null){
           this.mAffirmativeCallback.onPermissionConfirmed();
       }
   }
   //others
}

Then in activity, you could use like this way:

public class MainActivity extends AppCompatActivity {
    protected List<PermissionHelper> mPermissionHelpers = new ArrayList<>();
    
    @Override
    public void onRequestPermissionsResult(int requestCode,
                                             String permissions[], int[] grantResults) {
      for(PermissionHelper helper : mPermissionHelpers){
          helper.onRequestPermissionsResult(requestCode,permissions, grantResults);
      }
    }
    
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        mToolbar = (Toolbar) findViewById(R.id.toolbar);
        setSupportActionBar(mToolbar);
        
        PermissionHelper permissionHelper = PermissionHelper.permissionHelper(PermissionType.LOCATION, this,
                   new PermissionHelper.PermissionAffirmativeCallback() {
                       @Override
                       public void onPermissionConfirmed() {
                           renderMap();
                       }
                   });
        mPermissionHelpers.add(permissionHelper);
        
        permissionHelper = PermissionHelper.permissionHelper(PermissionType.CALL, this,
        new PermissionHelper.PermissionAffirmativeCallback() {
            @Override
            public void onPermissionConfirmed() {
                makeCall();
            }
        });
        mPermissionHelpers.add(permissionHelper);        
    }
}

There you go.

Full code on gist: https://gist.github.com/tuo/2ee5de2a03e04b48b79b

智利圣地亚哥Santiago-一个月生活琐记

2015-05-24T21:08:57+00:00

来智利圣地亚哥快一个月了，虽然我周一到周五都得上班，但是周末还是逛了逛市区，顺便去了趟瓦尔帕莱索valparaiso和比尼亚德尔马(Viña del Mar)或多或少有些了解生活各个方面的东西，有些有意思的事情还是值得写写的。

圣地亚哥

气候

我来的时候是四月三十号，也就是北半球快进入夏天，而这边刚好相反刚刚从盛夏进入秋天，温度大概23度左右。但是别看最高温度感觉还挺温暖，而且加上南半球（个人感觉，之前在墨尔本也是，阳光非常强烈，非常晒人）阳光强烈，中午的时候还有点热。但是早晚温差极大，早上冷的要死，中午有点热，智利同事告诉我穿衣服最好有点层次，这样可以及时调整，日出很晚，早上8点还是蒙蒙亮，当然日落也很晚咯，难怪后来我去瓦尔帕莱索valparaiso的路上有很多酒庄了。现在到了五月底，天气越来越冷了，早晚6-7度，中午18-19度,特别是早上冷的跟狗一样，我在<Culture Shock! Chile>这本书看到说有俄国人从智利回去后说智利的冬天才是世界上最冷的，我跟同事说起这样，他说这个还真是，因为智利对寒冷从来也没有什么准备，所以冬天家里也很超冷的那种。天气比较干燥，我来了快一个月，圣地亚哥也没有看到下雨，几乎每天都是阳光灿烂，非常明亮的那种，所以这边几乎每个人中午出去都有带墨镜或者帽子，其实并不只是为了装酷得。

圣地亚哥是一个很有特点的城市，周围被山脉包围，那种可以从家里可以看到远处的印第安山脉那种，周末登山干啥的很是平常。当然也不是每天都能看到了，因为圣地亚哥也有因为它的雾霾出名，有些时候，特别是秋冬天，可能就看不到远处的山脉，跟国内差不多那种。

货币

智利用的是比索，汇率这个记得不是很清楚，有一个简单的办法，就是去掉后面两个零，比如1000比索，就大概相当于10元。到这边阿图罗机场后，里面有一个货币兑换的地方，但是汇率比外面的要低，因为毕竟方便嘛。其实你可以在那里兑换一点点，然后到了市区，很多银行和购物中心都有货币兑换的地方，非常方便而且汇率更高点。

一罐可乐大概是500到750比索，相当于人命币5元到7元左右。物价还是比国内高很多的，在南美来看也是挺高的，我提到下面饮食的时候会更多提到的。

交通

智利比我想象的要发达多了，毕竟至少同事他家人去欧洲和美国都不需要办什么签证，这一点还是可以看出这个国家经济啥还是听不错的。圣地亚哥这个城市还是挺干净听整洁的，当然好的街区，比如我生活和工作的地方(Las Condes, Providencia), 就像西方那种，一个小花园，外面有个围栏，花园里几乎都摆着烧烤架（烧烤就跟我们吃火锅是一样的，家人朋友相聚在一起聊天啥），各种颜色的房子，当然了毕竟人家主要是西班牙后裔，所以风格几乎也是差不多。但是也有比较穷得地区，往往在城市的另一头，如果你做车从飞机场到市区，一路上你就可以看到那种比较穷的地方，然后慢慢的到了市区，整个环境社区都慢慢好了起来。

交通的话，这边用的跟我们那边差不多，也是公交车和地铁通用的交通卡，这边叫bip卡，因为你刷卡的时候，机器会“哔”的一声，表示已经扣钱了。但是在国内的话，公交车和地铁你得分开刷才行，这边的话，你只需付刷一次的钱，在两小时之内，你可以随意转乘公交地铁。一次这边费用大概是780比索，也就是7块8毛，比国内还是贵个三四块吧。智利地铁还是很方便的，但是不比国内的有啥屏蔽门，这边比较旧点，但是到站基本会报站名（没有英文，只有西班牙语），也没有国内地铁上LED提示站名那种，所以你需要特别注意到站时两侧的标识，别坐过了。

公交车嘛，绝大部分是红色，也有个别绿色的，条件没有国内的那么好，没有显示到了哪站了，下一站是什么那种语言和电子屏幕显示，所以需要特别注意别做过了，我一般都是打开google map或者Moovit 这两个应用盯着看，快到的时候，赶紧按旁边的到站按钮，可以告诉司机你想要到下一站下车，因为这边司机如果发现到站时没有人招手示意停车，就不会停靠那一站的。司机开车都非常猛，到站停车时候你可以要抓好扶手，不然直接人都甩出去了。

出租车，现在起步价大概是300比索，还是挺便宜的，而且很方便，我打过七八次车，基本上也有坑你的，但是绝大部分还是挺好挺不错的。

推荐使用Google Map或者Moovit, 虽然有人推荐http://www.transantiago.cl 但是那个只有西班牙语，没有英文这个还是挺麻烦的。

治安

整体上治安还是不错的，比不上上海啊啥的，但是在整个南美洲还是非常好的，特别是在好的街区，比如las condes和Providencia, 你基本上边走路边玩iPhone也都ok.需要说明的是，智利这边用iPhone的人不多，大部分还是安卓手机，所以我坐地铁时候就看到华为很大的广告牌，哈哈。但是如果你到了一些比较穷点的地方，还是要小心点，比如我之前去了这边的San Cristóbal Hill山（就在市区，从山上可以看到整个城市的全貌），同事提醒我需要注意财产安全，他们不会直接抢你，但是可能会偷走你得钱包啥，所以还是小心为上，不要露富。。。

其中一个周末我去了下这边比较有名的武器广场，这个还是比较乱得，因为智利的经济情况很不错，所以吸引了很多邻国的秘鲁人，其中有很多是非法过来的，基本上占据了这些有名的旅游景点。顺便推荐下武器广场旁边的前哥伦比亚博物馆，有很多关于南美和智利土著人的历史和文化介绍。

人民

南美人民还是很热情的，虽然讲英语的真的很少，所以如果你可以的话，尽量学点西班牙语过来，说起这个，我真是泪流满面~ 智利人有南美人那种奔放热情，家庭观念真的还很强的，每两个周末都有聚在一起烧烤啊，而且往往家庭成员非常多，大家一起边喝酒边吃肉，各种聊天。这边即使有长辈在，也不会介意长辈身份啥，各种嗨皮的开玩笑。见面有亲脸礼，这个我还是有点不习惯的。

我参加过好几次家庭或者朋友烧烤，肉吃的真是嗨皮啊，让我打开眼界，从未吃过这么多肉。

饮食

除了各种面包，三明治，汉堡，热狗之外，也有很多秘鲁菜馆，中国菜，意大利菜也有，但是最多还是日本菜，那个一个叫多啊，四处都是，基本上价格还是比较高得。

基本上我工作的地方在San Sebastian，地点还是非常好的，旁边也有各种餐馆。我比较常去的是旁边的当地餐馆Domino, 那里的热狗有很多总类，我已经基本上尝遍了各种，也有一个原因是便宜，20块一个左右。说到这里不得不提到这边非常常见的酱：牛油果酱(avocado). 几乎什么菜都能搭配，三明治，薯片，烧烤各种你能说出来的，都有。第一天到这边的晚上，同事他妈妈就做了一个三明治，里面一层的牛油果酱，别说味道还真是挺好吃的。在这里，基本就别想啥国内10块那种拉面啥了，基本上吃顿饭，大概四五十六七十块以上吧，比如subway,这边大概40快，而且味道实在不咋样。

去吃了办公室旁边的秘鲁菜，就是简单的一个小牛排加米饭薯条，还没点饮料，将近130块吧，后来去了跟同事整个妈妈妹妹老婆孩子姐姐去了远点的秘鲁菜，味道超级好，而且比这边还便宜。

这边我非常喜欢的特色食物就是：Empanada 也是这边国民小吃。我基本上都在这边的面包店Castaño买的，这家店在圣地亚哥几乎是满大街都是，就像上海的全家啊之类的，非常多，里面有卖面包，饮料等等。Empanada有两种: 一种类似于咱们的生煎包，外面是被煎炸过得，脆脆的；另一种，外面是就是咱们那种卷，酥酥软软的。里面最多是起司(queso)，或者火腿（jambo)等等，味道非常好吃，极力推荐。

这边菜虽然比国内要贵，但是普遍分量都还是挺大的。另外需要注意的是，普遍都是会给小费的，大概是10%吧。

智利最出名的就是红酒了，超市各种红酒，价格非常便宜，这边三四十快估计相当于国内百多块的酒了，非常好的酒。这边人也很喜欢喝pisco，度数更高点，所以很多人跟可乐混在一起，叫做Piscola, 味道还是不错的，但是后劲有点猛。

智利圣地亚哥Santiago-准备

2015-05-14T21:08:57+00:00

一年多以前我其中一个做开发的智利老板和一个做市场的智利同事经常中午出去吃饭聊天，我们就聊到了智利这个国家，毕竟南美这个大洲在我的印象是比较模糊的，应该是巴西上帝之城那个中疯狂，再加上我之前有朋友在巴西早上出门立马包就抢走了的经历，我觉得智利估计也是差不多的把。然后同事立马更正了我，他们说智利比巴西好很多，治安经济都很不错，他说你应该去体验下。当初我还有点不太相信，感觉好像南美就阿根廷可能比较好点，智利在我的脑海里就停留在超遥远的世界那端狭长的国土。因为我们经常吃兰州拉面，他开玩笑说你可以来圣地亚哥来开拉面馆啊

过了一阵子我智利做市场的同事回智利圣地亚哥去开拓市场，当然我们之间还联系，后来我想了想为什么不了，然后我跟公司说我能否去智利工作一阵子，因为我的工作性质（程序员）所以其实在智利工作跟在上海工作没有太大的区别，当然沟通成本稍微大点。老板们同意了，然后我开始了准备签证的事情

智利签证准备

本来准备想申请工作签证，这样可以一次可以待得久点，后来发现工作签证的要求实在太高了，双方公司都需要有什么法律证明，还有贸易往来的记录，超级麻烦。后来想干脆申请旅游签证，但是当我去智利领事馆咨询时（智利领事馆都不知道跑了多少遍，吐血啊，之前签证都是公司办的，我基本都没有自己动手，现在自己咨询和准备这些资料简直吐血），咨询处是一个满脸不爽的老太太，人家一点不好声好气的说：你要办什么签证，旅游啊，那只能给20天。我心理尼玛，不会吧，这么坑爹，然后回去后网上搜索了一番，跟智利同事商量，想到了一个好办法，就是申请类似于探亲访友的旅游签证 - 他去出具一份有智利外事部，公证处盖章的邀请函。我网上搜索一个模板，然后改了改，大概就是写明谁邀请谁去智利干什么事情，然后写明被邀请人的信息，待多少天。

整个西班牙系的殖民地都有西班牙的深刻烙印 - 慢。同事墨迹墨迹了好久，估计两三个月才搞好，本来预定过年之后三月一号出发的，结果人家二月七号邀请函才搞好寄过来，然后我只有过年放假之前的五天时间准备其他资料，包括买机票，去银行开了最近六个月的流水清单，五万块存款冻结证明，公司准假证明等等，赶紧周五最后一天跑到领事馆提交资料，大妈看了看看资料说这个不行啊，你得去北京办才行。我擦，我心里惊出一身冷汗，时间可是来不及啊，回头一想，同事提醒我你要带好居住证才行，幸亏我之前很早就办好了，一切都顺利。等了半个小时，大妈出来了说按照邀请函上的时间，批了你九十天多次出入境，也就是旅游签证能批下来最多的天数，你三月二十四来取吧。

然后是脑袋进水，当初选航线的时候，我智利同事一般都飞智利-加拿大-中国这个航线，但是当同事跟我去加拿大航空官方网站去搜索时，发现这条航线根本就没有，也不知道是什么原因 @_@ 另外就是飞上海-巴黎-圣地亚哥，但是这条航线问题就是贵，贵了不知一两千，而是四五千快，这两条应该是比较普遍的路线了。我转而看看有没有其他选择，就发现了英国航空有从上海-伦敦-马德里-圣地亚哥的路线，我一想多点中转站也没关系，还可以多休息下也挺好，然后价格便宜啊，于是就定了。

但是我等我过完年回上海，发现英国是需要过境签证的，哪怕你只是转机，我看了官方网站确实是需要，然后我就开始准备英国的过境签证申请，但是我发现我需要先拿到智利批下来的签证才能申请，而智利签证是在三月二十四号才批下来，意味着只有五天的时间来提交申请，预约时间，提交材料等等，根本来不及。于是我又去了熟悉了智利领事馆说能不能加急下，好说歹说，提前了五天，也就是三月十九号。尼玛干捉急也没办法，还是等着吧。

吐血的英国过境签证

到了三月十九号，拿到了智利签证，我开始了填写英国过境签证在线表格，尼玛好多东西需要填写，你的个人资料，背景，收入情况，家庭情况，然后是目的地国家签证，我当时填的比较赶，有一个地方填错了就是经转目的地，我写了有目的地国家签证，但是我填的是智利签证，然后就提交上去了，然后预约时间，去了英国签证中心，发现时间来不及，必须加急（五天），一千块费用，也管不上了，录入指纹照片等等搞下来，最后说你大概下周一也就是三月三十号来取，但是我是三十一号的飞机。惴惴不安的等到了三十号，我在领事馆外面等了两小时，人家说签证被拒了，说是人家说我不能证明我可以进入西班牙。尼玛，出发前最后一天发现不行不能出去，那是一种什么感觉。而且我房子也是刚好月底到期，悲剧啊，尼玛漂泊在外了。回去距离飞机起飞大概不到12小时，赶紧打电话给英国航空客服改签了航班，改签费一千五白块:(，当时没来的及（时间很紧张），脑残的改签到了一个星期后，后来发现尼玛根本不可能赶上的。

得了，我赶紧跟老板们商量了下，也打过电话也英国签证咨询顾问，本来想让英国老板（三个老板之一）去打电话给英国领事馆看看会不会好点，但是一想估计也没戏，最后决定写一封抗议信也就是申诉信（当然申诉这种事情，人家可能会花很长时间才回复你，总之不要抱任何希望），简单描述了我被拒的原因，写上了关于西班牙转机不需要签证的这个情况。关于西班牙转机是否需要签证的这个问题，尼玛我要吐槽下，首先官方网站上明确没有写明，也就是24小时内国际区域内转机是不需要过境签证的这条。然后我赶紧联系上上海西班牙签证中心，打电话说明了我的情况，问人家我到底需不需要什么过境签证了，人家说这个她得问一下签证官，第二天回电话给我。到了第二天，没有任何电话，我赶紧下班之前回拨过去，还在说他们在确认中。我擦，心想尼玛上海签证中心太不靠谱，因为他们下午4点就下班，我想不行得想其他办法，我一看广州西班牙签证中心是五点下班，于是我打电话给广州总部，人家明确告诉我你不需要过境签证，没有问题，直接过去转机就ok,我当时觉得豁然开朗，虽然网上也是这么说，但是能从领事馆那里得到肯定的回答还是很安心的。

刚好我上次改签的时间到了，于是只能是再改签一次，这次改签到了四月底，这样一来能够有足够的缓冲时间来办签证，当时航班起飞前一天，碰巧中国这边放假了，尼玛，只能打电话给英航的英国总部，但是那个电话需要你设置信用卡什么乱七八糟的，很麻烦。弄好了之后，赶紧让他帮忙改签，结果那边电脑碰巧坏了，都只能是人工去查，最后查出来改签到四月底需要五千块，尼玛坑爹啊，上次只有一千五，好在老板说给报销了，然后报了我的信用卡去扣费，挂了电话觉得这下好了，也算是搞定了。结果一晚上好像手机都没有收到扣费短信，我觉得有点不对，第二天一大早上，英航客服上午九点上班，我的飞机是上午十一点半，然后我赶紧打电话让她查询下，果然是扣费失败，于是我让她在扣一次，手机信息提示额度不够，赶紧要临时提高额度，总算是有惊无险的搞定了改签。

傻逼的西班牙驻上海签证中心

这下我决定，这次一定要好好准备，不可以仓促了。于是中间过去这一周，上海西班牙领事馆始终没有回我电话，终于过了一周之后人家回复我了，说你这个情况，签证官说是需要办理过境签证的，我说不对啊，我咨询了广州西班牙签证中心，人家说我这个情况是不需要的啊，她冷冷的说，这里是上海，跟广州不一样，你也可以不听我们的，但是如果你那时候在机场被截住，你就需要自己负责了，你自己看着办吧。挂了电话，我心理万头草泥马奔过，法克法克法克法克！！！

于是我要赶紧打电话给西班牙总领事馆，再次确认，人家说确实不需要，但是我们留了个心，我用中英文把我的航班信息贴出来，然后发邮件给广州领事馆咨询是否需要过境签证，然后将邮件往来都打印了出来（事实证明这个非常重要，以后会提到）。然后重新填写英国过境签证申请表，跟英国和美国同事将每一个条目仔细过了一遍，确认没有问题，这一次我直接说我没有西班牙的签证。最后提交资料的时候，我将邮件来往的打印件一并提交。

在受理之后的两天，我收到了英国签证中心的邮件，他回复我上次的申诉：

Dear Huang Tuo,

Thank you for your below correspondence.
I note that you have since re-applied for your DATV and have addressed the reasons for refusal, therefore have now been issued.
For any future enquiries regarding UK visas, please refer to our website at https://www.gov.uk/browse/visas-immigration

Regards

I. Keating
Entry Clearance Manager
Shanghai Visa Section
UK Visas and Immigration

还是挺快的嘛，我放心了，顺利拿到了英国的过境签证，但是我心理还回荡着上海西班牙签证那冷冷的回复，还是心里面有点点不安，虽然我知道应该是不用的，但是这上海跟广州两处说法不一样，真的是令人很无语，很纠结，很操蛋~之前去印度和墨尔本都是直达的，从未有过这次转机，还是转了两次，尼玛真的是令人欲哭无泪

总之，我就安心准备各种材料，给同事买的七八包他喜欢吃的黄飞红麻辣花生，淘宝来的macbook电源，给他老婆的，他女儿，他妹妹，他妈妈的礼物，各种淘宝吧。毕竟我去那边，可能会住在他妈妈家。

惊险旅途

到了四月二十九号，如约出发，早上四点起床，定了四点半的出租车到了浦东机场，等到了五点多排队去办理登记手续，我给英航的工作人员看了英国的过境签证，智利的签证，人家说诶，你有没有西班牙的，我说的没有啊，但是我有跟领事馆确认的邮件来往打印件，于是他好像跟他们领导还是什么确认了好一会，最后说去申根国应该是需要的，但是你这个情况应该没问题吧，给你办登记手续。但是登机牌只打出来上海和伦敦的，但是伦敦的登机牌并没有登机口，因为暂时还不知道，跟别说从西班牙到智利的登机牌，那个都没有打出来，她说你到了西班牙机场再去找工作人员要登机牌吧，我心想嚓，好吧。行李箱刚好22.9公斤，规定的重量是23公斤，好险。

我在伦敦希斯罗机场转机只有两个小时的时间，两个小时的时间内我需要办理过境签证检查手续，安检，查询那个登机口。出发前我去希斯罗机场的官网上注册了一个账号，因为希斯罗机场有免费wifi，但是有时间限制，看网上说只有45分钟，我想我两小时，赶紧注册个账号这样可以拿到double时间，但是实际到了机场发现其实免费的你可以用4个小时，足够了。从上海飞伦敦预计发行12小时，尼玛结果飞了13个小时，这样一来我再希斯罗机场转机的时间就非常非常紧张了。下了飞机，我一路狂奔，转乘摇摆地铁，到了转机边检的地方，我一看我擦，这么人多排队，我赶紧跟工作人员我说我快赶不上飞机了，有没有快速通道（express), 她赶紧带我过去，人家看了过境签证跟我说你赶紧跑，以最快的速度跑，快快快！我心想我也想啊，但是一去安检人也很多而且速度还慢，我心里焦急但是不得不安慰自己，要淡定。过安检时，真是检查非常仔细，费了不好时间，好在通过了，赶紧去看登机口信息，然后速度飞奔到登机口，赶在最后一通登记提醒时赶上了去马德里的飞机。

一路有惊无险的到了马德里机场，我之前的心里一路上都是惴惴不安的，因为我担心会不会被截住说你需要有什么签证，但是要安慰自己我有邮件往来，而且应该没有问题，那种感觉还真是挺纠结的，不过出乎我意料，马德里机场相当的松，我只是简单的过个安检，顺利进入了登记区域，我瞬间觉得轻松了，感觉这一路从申请签证到成行是那么的坎坷那么不容易，终于我可以确认我就差十几个小时就可以到智利了。机场免费WiFi是三十分钟，于是我买了一个小时也就是五欧元的流量，尼玛也听坑爹的，不过赶紧报了平安，等了五个小时，到了晚上十二点多，终于开始登机了。

经过十三个小时的飞行，斜飞了整个大西洋终于到了智利圣地亚哥。出乎意料，安检都很顺利的出了机场，同事已经在机场外面等着了，顺利看到了他，然后坐车去了他妈妈家，他妈妈妹妹都上班去了，我赶紧坐在院子里面休息喝了点水，心理想着：

尼玛，

南美，终于来了；圣地亚哥，来了~~~

GPUImage Video Merge: Fix Movie Writer

2015-02-05T13:57:39+00:00

I have been used GPUImage to do some chroma key video merge(I extended GPUImageThreeInputFilter to support four videos blending). But when I try to use GPUImageMovieWriter to write out blended videos, I found some problems - typically those are very flaky to reproduce and it just happens from time to time, which causes lots of headache.

Typical problems/limits you might run into when use GPUImageMovieWriter are:

Output video got cut off Prematurely, i.e, last one second lost #1403, #1635
Only one audio track got written out #934
Frame dropping, frame time out of sync, some video frame freezes #1501, #744
Video writing ends before audio’s, which last one or two seconds video freeze #830
Black frames at the beginning #52, #652
Poor stability, random crashes #773, #1634

I already have elaborated on solutions to those problems in my blog GPUImage Video Merging: Timestamp Synchronization and GPUImage Movie Writer: Merging All Audio Tracks From Multiple Movies.

But I need a way to incorporate those changes to library nicely without completely writing from scratch so that I could facilitate existing functions of GPUImage.

What we will do is create three classes: THImageMovie, THImageMovieWriter and THImageMovieManager.

To start, let’s duplicate GPUImageMovie and GPUImageMovieWriter and name it to THImageMovie and THImageMovieWriter.

THImageMovie

In THImageMovie, we’re not gonna do many changes:

- (void)processAsset
{
    reader = [self createAssetReader];


    AVAssetReaderOutput *readerAudioTrackOutput = nil;

    audioEncodingIsFinished = YES;
    for( AVAssetReaderOutput *output in reader.outputs ) {
        if( [output.mediaType isEqualToString:AVMediaTypeAudio] ) {
            audioEncodingIsFinished = NO;
            readerAudioTrackOutput = output;
        }
        else if( [output.mediaType isEqualToString:AVMediaTypeVideo] ) {
            readerVideoTrackOutput = output;
        }
    }

    if ([reader startReading] == NO) 
    {
            NSLog(@"Error reading from file at URL: %@", self.url);
        return;
    }

    //============================We delete all the code left, just signify that I'm done with asset warming up, ready to be processed
	dispatch_group_leave([THImageMovieManager shared].readingAllReadyDispatchGroup);
}

- (BOOL)renderNextFrame {
    __unsafe_unretained THImageMovie *weakSelf = self;
    if (reader.status == AVAssetReaderStatusReading && (!_shouldRepeat || keepLooping))
    {

        return [weakSelf readNextVideoFrameFromOutput:readerVideoTrackOutput];
    }

    if (reader.status == AVAssetWriterStatusCompleted) {
        NSLog(@"movie: %@ reading is done", self.url.lastPathComponent);
        [reader cancelReading];
        [weakSelf endProcessing];
    }

    return NO;
}

Above is the most important changes, we basically remove readNextVideoFrameFromOutput in processAsset method, and add a new method - (BOOL)renderNextFrame.

What we do is renderNextFrame is just read next video frame and return YES/NO if reading is completed. One important thing here we dont’ call endProcessing in readNextVideoFrameFromOutput anymore, you could check in detail commit on github.

This is pretty much the changes for movie class.

THImageMovieWriter

In header class THImageMovieWriter.h:

//add one more parameter movies to init method
- (id)initWithMovieURL:(NSURL *)newMovieURL size:(CGSize)newSize movies:(NSArray *)movies;

//remove the instance variable assetWriterPixelBufferInput and move it as property to prevent memory leak when used in block 
@property(nonatomic, retain) AVAssetWriterInputPixelBufferAdaptor *assetWriterPixelBufferInput;

in THImageMovieWriter.m, First, let’s add multiple audio tracks reader using AVAssetReaderAudioMixOutput and writer:

#pragma mark setupAssetWriter
- (void)setupAudioAssetReader {

    NSMutableArray *audioTracks = [NSMutableArray array];

    for(THImageMovie *movie in self.movies){
        AVAsset *asset = movie.asset;
        if(asset){
            NSArray *_audioTracks = [asset tracksWithMediaType:AVMediaTypeAudio];
            if(_audioTracks.count > 0){
                [audioTracks addObject:_audioTracks.firstObject];
            }
        }
    }
    AVMutableComposition* mixComposition = [AVMutableComposition composition];
    for(AVAssetTrack *track in audioTracks){
        if(![track isKindOfClass:[NSNull class]]){
            AVMutableCompositionTrack *compositionCommentaryTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeAudio

                                                                                                preferredTrackID:kCMPersistentTrackID_Invalid];
            [compositionCommentaryTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, track.asset.duration)
                                                ofTrack:track
                                                 atTime:kCMTimeZero error:nil];
        }
    }

    self.assetAudioReader = [AVAssetReader assetReaderWithAsset:mixComposition error:nil];
    self.assetAudioReaderTrackOutput =
            [[AVAssetReaderAudioMixOutput alloc] initWithAudioTracks:[mixComposition tracksWithMediaType:AVMediaTypeAudio]
                                                       audioSettings:nil];

    [self.assetAudioReader addOutput:self.assetAudioReaderTrackOutput];
}

- (void)setupAudioAssetWriter{
    double preferredHardwareSampleRate = [[AVAudioSession sharedInstance] currentHardwareSampleRate];
    AudioChannelLayout acl;
    bzero( &acl, sizeof(acl));
    acl.mChannelLayoutTag = kAudioChannelLayoutTag_Mono;

    NSDictionary *audioOutputSettings = [NSDictionary dictionaryWithObjectsAndKeys:
            [ NSNumber numberWithInt: kAudioFormatMPEG4AAC], AVFormatIDKey,
            [ NSNumber numberWithInt: 1 ], AVNumberOfChannelsKey,
            [ NSNumber numberWithFloat: preferredHardwareSampleRate ], AVSampleRateKey,
            [ NSData dataWithBytes: &acl length: sizeof( acl ) ], AVChannelLayoutKey,
            //[ NSNumber numberWithInt:AVAudioQualityLow], AVEncoderAudioQualityKey,
            [ NSNumber numberWithInt: 64000 ], AVEncoderBitRateKey,
                    nil];

    assetWriterAudioInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeAudio outputSettings:audioOutputSettings];
    [assetWriter addInput:assetWriterAudioInput];
    assetWriterAudioInput.expectsMediaDataInRealTime = _encodingLiveVideo;
}

Next, change startRecording to following:

- (void)startRecording;
{
    dispatch_group_notify([THImageMovieManager shared].readingAllReadyDispatchGroup, dispatch_get_main_queue(), ^{
        NSLog(@"all set, readers and writer both are ready");
        [self setupAudioAssetReader];
        [self setupAudioAssetWriter];

        alreadyFinishedRecording = NO;
        isRecording = YES;

        BOOL aduioReaderStartSuccess = [self.assetAudioReader startReading];
        if(!aduioReaderStartSuccess){
            NSLog(@"asset audio reader start reading failed: %@", self.assetAudioReader.error);
            return;
        }

        startTime = kCMTimeInvalid;
        [self.assetWriter startWriting];
        [self.assetWriter startSessionAtSourceTime:kCMTimeZero];
        NSLog(@"asset write is good to write...");

        [self kickoffRecording];
    });
}

- (void)kickoffRecording {
    // If the asset reader and writer both started successfully, create the dispatch group where the reencoding will take place and start a sample-writing session.
    self.recordingDispatchGroup = dispatch_group_create();
    self.audioFinished = NO;
    self.videoFinished = NO;

    [self kickOffAudioWriting];
    [self kickOffVideoWriting];

    __unsafe_unretained typeof(self) weakSelf = self;
    // Set up the notification that the dispatch group will send when the audio and video work have both finished.
    dispatch_group_notify(self.recordingDispatchGroup, [THImageMovieManager shared].mainSerializationQueue, ^{
        weakSelf.videoFinished = NO;
        weakSelf.audioFinished = NO;
        [weakSelf.assetWriter finishWritingWithCompletionHandler:^{
            if(weakSelf.completionBlock){
                weakSelf.completionBlock();
            }
        }];
    });
}

Here we only start process when all movies are ready and use dispatch_group to ensure audio and video both finished.

- (void)kickOffAudioWriting{
    dispatch_group_enter(self.recordingDispatchGroup);
    __unsafe_unretained typeof(self) weakSelf = self;

    CMTime shortestDuration = kCMTimeInvalid;
    for(THImageMovie *movie in self.movies) {
        AVAsset *asset = movie.asset;
        if(CMTIME_IS_INVALID(shortestDuration)){
            shortestDuration = asset.duration;
        }else{

            if(CMTimeCompare(asset.duration, shortestDuration) == -1){
                shortestDuration = asset.duration;
            }
        }
    }
	
    [assetWriterAudioInput requestMediaDataWhenReadyOnQueue:[THImageMovieManager shared].rwAudioSerializationQueue usingBlock:^{
        // Because the block is called asynchronously, check to see whether its task is complete.
        if (self.audioFinished)
            return;

        BOOL completedOrFailed = NO;
        // If the task isn't complete yet, make sure that the input is actually ready for more media data.
        while ([assetWriterAudioInput isReadyForMoreMediaData] && !completedOrFailed) {
            // Get the next audio sample buffer, and append it to the output file.
            CMSampleBufferRef sampleBuffer = [self.assetAudioReaderTrackOutput copyNextSampleBuffer];
            if (sampleBuffer != NULL) {
                CMTime currentSampleTime = CMSampleBufferGetOutputPresentationTimeStamp(sampleBuffer);
                BOOL isDone = CMTimeCompare(shortestDuration, currentSampleTime) == -1;
				
                BOOL success = [assetWriterAudioInput appendSampleBuffer:sampleBuffer];
                weakSelf.audioWroteDuration = CMTimeGetSeconds(currentSampleTime);
                if (success) {
                    //NSLog(@"append audio buffer success");
                } else {
                    NSLog(@"append audio buffer failed");
                }
                CFRelease(sampleBuffer);
                sampleBuffer = NULL;
                completedOrFailed = !success;

                if(isDone){                   
                    completedOrFailed = YES;
                }
            }
            else {
                completedOrFailed = YES;
            }
        }//end of loop

        if (completedOrFailed) {
            NSLog(@"kickOffAudioWriting wrint done");
            // Mark the input as finished, but only if we haven't already done so, and then leave the dispatch group (since the audio work has finished).
            BOOL oldFinished = self.audioFinished;
            self.audioFinished = YES;
            if (!oldFinished) {
                [assetWriterAudioInput markAsFinished];
                dispatch_group_leave(self.recordingDispatchGroup);
            };
        }
    }];
}

in audio writng part, we got shortest duration of video and prevent final audio duration in output is longer than video.

- (void)kickOffVideoWriting {
    dispatch_group_enter(self.recordingDispatchGroup);
    self.isFrameRecieved = NO;
   __unsafe_unretained typeof(self) weakSelf = self;
    self.firstVideoFrameTime = -1;
    self.onFramePixelBufferReceived = ^(CMTime frameTime, CVPixelBufferRef pixel_buffer){        
        [weakSelf.assetWriterPixelBufferInput appendPixelBuffer:pixel_buffer withPresentationTime:frameTime];
        if(weakSelf.firstVideoFrameTime == -1){
            weakSelf.firstVideoFrameTime = CMTimeGetSeconds(frameTime);
        }
        CVPixelBufferUnlockBaseAddress(pixel_buffer, 0);
        weakSelf.videoWroteDuration = CMTimeGetSeconds(frameTime);
        if (![GPUImageContext supportsFastTextureUpload])
        {
            CVPixelBufferRelease(pixel_buffer);
        }
        weakSelf.isFrameRecieved = NO;
    };

    [assetWriterVideoInput requestMediaDataWhenReadyOnQueue:[THImageMovieManager shared].rwVideoSerializationQueue usingBlock:^{
        if (self.videoFinished)
            return;
        BOOL completedOrFailed = NO;
        // If the task isn't complete yet, make sure that the input is actually ready for more media data.
        while ([assetWriterVideoInput isReadyForMoreMediaData] && !completedOrFailed) {
            if(!self.isFrameRecieved){
                self.isFrameRecieved = YES;
                for(THImageMovie *movie in self.movies){
                    BOOL hasMoreFrame = [movie renderNextFrame];
                    //NSLog(@"--movie: %@, has more frames: %d", movie.url.lastPathComponent, hasMoreFrame);
                    if(!hasMoreFrame){
                        completedOrFailed = YES;
                        break;
                    }
                }
            }
        }

        if(completedOrFailed){
            NSLog(@"kickOffVideoWriting mark as finish");
            // Mark the input as finished, but only if we haven't already done so, and then leave the dispatch group (since the video work has finished).
            BOOL oldFinished = self.videoFinished;
            self.videoFinished = YES;
            if (!oldFinished) {
                for(THImageMovie *movie in self.movies){
                    [movie cancelProcessing];
                }
                [assetWriterVideoInput markAsFinished];
                dispatch_group_leave(self.recordingDispatchGroup);
            }
        }
    }];
}

Here in video writing, we defined onFramePixelBufferReceived which should be called when final rendered result in GPU is ready in newFrameReadyAtTime. Only if we have recieved merged pixels then we kick off next video reading round. So in this way, we could ensure the frame-perfect video writing and fully synced timestamps.

- (void)newFrameReadyAtTime:(CMTime)frameTime atIndex:(NSInteger)textureIndex;
{
    if (!isRecording)
    {
        [firstInputFramebuffer unlock];
        return;
    }

//    // Drop frames forced by images and other things with no time constants
//    // Also, if two consecutive times with the same value are added to the movie, it aborts recording, so I bail on that case
//    if ( (CMTIME_IS_INVALID(frameTime)) || (CMTIME_COMPARE_INLINE(frameTime, ==, previousFrameTime)) || (CMTIME_IS_INDEFINITE(frameTime)) )
//    {
//        [firstInputFramebuffer unlock];
//        return;
//    }
//
//    if (CMTIME_IS_INVALID(startTime))
//    {
//        runSynchronouslyOnContextQueue(_movieWriterContext, ^{
//            if ((videoInputReadyCallback == NULL) && (assetWriter.status != AVAssetWriterStatusWriting))
//            {
//                [assetWriter startWriting];
//            }
//
//            [assetWriter startSessionAtSourceTime:frameTime];
//            startTime = frameTime;
//        });
//    }

    GPUImageFramebuffer *inputFramebufferForBlock = firstInputFramebuffer;
    glFinish();

    [_movieWriterContext useAsCurrentContext];

    [self renderAtInternalSizeUsingFramebuffer:inputFramebufferForBlock];

    CVPixelBufferRef pixel_buffer = NULL;

    if ([GPUImageContext supportsFastTextureUpload])
    {
        pixel_buffer = renderTarget;
        CVPixelBufferLockBaseAddress(pixel_buffer, 0);
    }
    else
    {
        CVReturn status = CVPixelBufferPoolCreatePixelBuffer (NULL, [self.assetWriterPixelBufferInput pixelBufferPool], &pixel_buffer);
        if ((pixel_buffer == NULL) || (status != kCVReturnSuccess))
        {
            CVPixelBufferRelease(pixel_buffer);
            return;
        }
        else
        {
            CVPixelBufferLockBaseAddress(pixel_buffer, 0);

            GLubyte *pixelBufferData = (GLubyte *)CVPixelBufferGetBaseAddress(pixel_buffer);
            glReadPixels(0, 0, videoSize.width, videoSize.height, GL_RGBA, GL_UNSIGNED_BYTE, pixelBufferData);
        }
    }



    runAsynchronouslyOnContextQueue(_movieWriterContext, ^{
        if (!assetWriterVideoInput.readyForMoreMediaData && _encodingLiveVideo)
        {
            [inputFramebufferForBlock unlock];
            NSLog(@"1: Had to drop a video frame: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, frameTime)));
            return;
        }
        
        // Render the frame with swizzled colors, so that they can be uploaded quickly as BGRA frames
        [_movieWriterContext useAsCurrentContext];
        [self renderAtInternalSizeUsingFramebuffer:inputFramebufferForBlock];
        
        CVPixelBufferRef pixel_buffer = NULL;
        
        if ([GPUImageContext supportsFastTextureUpload])
        {
            pixel_buffer = renderTarget;
            CVPixelBufferLockBaseAddress(pixel_buffer, 0);
        }
        else
        {
            CVReturn status = CVPixelBufferPoolCreatePixelBuffer (NULL, [self.assetWriterPixelBufferInput pixelBufferPool], &pixel_buffer);
            if ((pixel_buffer == NULL) || (status != kCVReturnSuccess))
            {
                CVPixelBufferRelease(pixel_buffer);
                return;
            }
            else
            {
                CVPixelBufferLockBaseAddress(pixel_buffer, 0);
                
                GLubyte *pixelBufferData = (GLubyte *)CVPixelBufferGetBaseAddress(pixel_buffer);
                glReadPixels(0, 0, videoSize.width, videoSize.height, GL_RGBA, GL_UNSIGNED_BYTE, pixelBufferData);
            }
        }

        if(self.onFramePixelBufferReceived){
            self.onFramePixelBufferReceived(frameTime, pixel_buffer);
        }
        
        [inputFramebufferForBlock unlock];
    });
}

In newFrameReadyAtTime, we remove the asset write append pixel buffer part and call onFramePixelBufferReceived to notify asset writer to continue next round frame processing.

For THImageMovieManager, we just create some serial queue for video and audio writing, nothing special there.

We’re all good but how to give it run.

Demo Run(Chroma Key)

self.thMovieA = [[THImageMovie alloc] initWithURL:rawVideoURL];
self.thMovieFX = [[THImageMovie alloc] initWithURL:fxURL];
self.filter = [[GPUImageChromaKeyBlendFilter alloc] init];
[self.thMovieFX addTarget:self.filter];
[self.thMovieA addTarget:self.filter];

NSArray *thMovies = @[self.thMovieA, self.thMovieFX];
//setup writer
NSString *pathToMovie = [NSHomeDirectory() stringByAppendingPathComponent:@"Documents/th_output.mov"];
unlink([pathToMovie UTF8String]); // If a file already exists, AVAssetWriter won't let you record new frames, so delete the old movie
NSURL *outputURL = [NSURL fileURLWithPath:pathToMovie];
self.thMovieWriter =  [[THImageMovieWriter alloc] initWithMovieURL:outputURL size:CGSizeMake(640.0, 640.0) movies:thMovies];
[self.filter addTarget:self.thMovieWriter];

//---comment those audio one's
//self.movieWriter.shouldPassthroughAudio = YES;
//self.gpuMovieA.audioEncodingTarget = self.movieWriter;
//[self.gpuMovieA enableSynchronizedEncodingUsingMovieWriter:self.movieWriter];

[self.thMovieA startProcessing];
[self.thMovieFX startProcessing];
[self.thMovieWriter startRecording];

__weak typeof(self) weakSelf = self;
[self.thMovieWriter setCompletionBlock:^{
    [weakSelf.thMovieFX endProcessing];
    [weakSelf.thMovieA endProcessing];

	//no need to call finishRecording 
    NSLog(@"movie writing done");
}];

There you go.

I found this change has resolved all problems above and it runs quite stably(not even with a single crash spotted)

The commit is available on github: Fix movie writer exporting and you could run the sample project called ChromaKeyVideoMerge

Nodejs CronJob on Heroku with Scheduler

2015-01-27T12:20:08+00:00

Using nodejs with cronjob module to send daily email in midnight is quite easy. But when your app is running on heroku and you only have 1 web dyno, you probably could run into some weird things. For example, you could have nothing happen when pre-assumed cronjob should be fired; or sometimes it works as you expect and sometimes it wouldn’t.

Demo Case: Send Daily Emails

	

var cronJob = require('cron').CronJob;
var _cronJobs = {};

var createDailyMail = function(callback){
    //...
    console.log("====createDailyMail email");    
}
    
//remember to add 'time' dependency in package.json if you want to 'timeZone' feature
var _jobs = [
  {
    name: 'Create Email Queue',
	cronTime: '0 2 0 * * *', // every day at 00:02:00 am (extra 2 minutes for dyno wakeup, any number between 0 and 60 minutes is fine as cronjob will be guaranteed to be executed before dyno went sleep again)
    //cronTime: '*/10 * * * * *', // every 10 seconds
    onTick: createDailyMail,
    start: true,
    id: 'createdailyemail',
    timeZone: 'Asia/Shanghai'
  }
];


exports.schedule = function() {
	_jobs.map(function(job) {
		_cronJobs[job.id] = new cronJob(job);
		console.log(common.util.format('%s cronjob scheduled at %s on timezone %s',job.name, job.cronTime, job.timeZone));
	});
}

So you would just expect that it would print “createDailyMail” at 00:02:00 everyday midnight on heroku.

But it doesn’t follow your assumption everytime.

Why?

Dyno Sleeping

The reason is because web dyno went to sleep after some idle time to give you more free time to reduce cost you would pay for.

To verify that, fire your terminal and run heroku logs -n 1500, 1500 means how many lines of logs you want pull out to your local terminal console:

2015-01-23T05:02:00.567188+00:00 app[web.1]: emails: 0
2015-01-23T05:02:00.567197+00:00 app[web.1]: mailer.process_queue => Finished processing each email address from 0 queued emails
2015-01-23T05:02:48.546326+00:00 heroku[web.1]: Idling
2015-01-23T05:02:48.546730+00:00 heroku[web.1]: State changed from up to down
2015-01-23T05:02:50.845020+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2015-01-23T05:02:52.249230+00:00 heroku[web.1]: Process exited with status 143
2015-01-24T01:01:02.745463+00:00 heroku[web.1]: Unidling
2015-01-24T01:01:02.746350+00:00 heroku[web.1]: State changed from down to starting
2015-01-24T01:01:06.235129+00:00 heroku[web.1]: Starting process with command `node server.js`
2015-01-24T01:01:07.039014+00:00 app[web.1]: === LOGS ==========================================    
2015-01-24T01:01:09.412395+00:00 app[web.1]:  Create Email Queue cronjob scheduled at 0 2 0 * * * on timezone Asia/Shanghai
2015-01-24T01:07:24.702069+00:00 app[web.1]: Thu Jan 24 2015 01:07:24 GMT+0000 (UTC) | XXXX app started on port 27646
2015-01-24T01:07:25.729917+00:00 heroku[router]: at=info method=GET path="/" host=XXX.herokuapp.com request_id=a013d631-2bef-4b30-9314-4f356d7c7a8a dyno=web.1 connect=2ms service=18ms status=401 bytes=197
2015-01-24T01:07:25.728315+00:00 app[web.1]: GET / 401 - - 6.927 ms
2015-01-24T01:07:24.830608+00:00 heroku[web.1]: State changed from starting to up
2015-01-24T01:08:00.746585+00:00 app[web.1]: mailer.process_queue => Finished processing each email address from 0 queued emails
2015-01-24T01:08:00.746266+00:00 app[web.1]: emails: 0  

Our cron job fire time is 00:02:00 Shanghai time, which converted to UTC is “16:02:00”. But from above logs, you will notice that web dyno went sleep around 01:00:00 am and woke up around next day when some request hit it. So during this sleep time, it wouldn’t process any task at all including our cron job task who fired at 16pm.

Heroku has a blog on this topic: App Sleeping on Heroku, which mentioned:

When an app on Heroku has only one web dyno and that dyno doesn’t receive any traffic in 1 hour, the dyno goes to sleep.

And when it would wake up:

When someone accesses the app, the dyno manager will automatically wake up the web dyno to run the web process type. This causes a short delay for this first request, but subsequent requests will perform normally. Apps that have more than 1 web dyno running never go to sleep and worker dynos (or other process types) are never put to sleep.

We need to take a difference approach on this.

Heroku Scheduler

Since the heroku would shutdown the dyno when within one hour adn there is no request for it, and our cronjob only fires around midnight everyday, we could write a script that simply make a http request just before the cronjob fire time.

Tricky part is that we could put this script locally on some server, and setup another cronjob(meta cronjob) to fire this script. But this is not a good idea and just is over-kill for this simple task.

Luckily, heroku is aware of this problem and provide an adds-on called Heroku Scheduler.

It is a great fit for our case, and it is easy to setup and almost free because of what it is built on : the One-Off Dynos:

Scheduler add-on runs one-off dynos that will count toward your dyno-hours that you will be charged for each month.

Think about the price, heroku already provide you 750 free dyno-hours per app per month. Suppose my web dyno runs 23 hours per day then we still got 750 - 23 * 30 = 60 hours free time, and our Scheduler will get this free time.

Good news is that our scheduler is so simple, i.e. just make a http request and only runs once per day, so it is pretty much free for us :)

We start with the script by creating a file wake_up_dyno.js under the root of project:

var http = require('http'); //importing http

var options = {
    host: 'XXX.herokuapp.com',
    port: 80,
    path: '/WAKEUP_DYNO'
};
console.log("======WAKUP DYNO START");
http.get(options, function(res) {
    res.on('data', function(chunk) {
        try {
            // optional logging... disable after it's working
            console.log("======WAKUP DYNO: HEROKU RESPONSE: " + chunk);
        } catch (err) {
            console.log(err.message);
        }
    });
}).on('error', function(err) {
    console.log("Error: " + err.message);
});

Very straightforward.

Then we need to go to heroku scheduler adds-on page:

create a schedule job
set task name to node wake_up_dyno.js
dyno size 1x by default
frequence to daily
run time set to: 16:00 UTC
save

Then you’re pretty much done. Just pay attention to the run time, it is utc time so make sure you calculate/convert correctly from your target timezone. I have just made a dumb mistake by putting wrong 04:00 UTC time rather than 16:00 UTC.

Run it again and check out the logs:

2015-01-27T09:22:00.688992+00:00 app[web.1]: emails: 0
2015-01-27T09:22:00.689005+00:00 app[web.1]: mailer.process_queue => Finished processing each email address from 0 queued emails
2015-01-27T09:22:33.117304+00:00 heroku[web.1]: Idling
2015-01-27T09:22:33.117873+00:00 heroku[web.1]: State changed from up to down
2015-01-27T09:22:35.571568+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2015-01-27T09:22:37.153548+00:00 heroku[web.1]: Process exited with status 143
2015-01-27T16:00:13.589599+00:00 heroku[api]: Starting process with command `node wake_up_dyno.js` by scheduler@addons.heroku.com
2015-01-27T16:00:17.036490+00:00 heroku[scheduler.1203]: Starting process with command `node wake_up_dyno.js`
2015-01-27T16:00:17.661659+00:00 heroku[scheduler.1203]: State changed from starting to up
2015-01-27T16:00:18.793986+00:00 app[scheduler.1203]: ======WAKUP DYNO START
2015-01-27T16:00:18.868482+00:00 heroku[web.1]: State changed from down to starting
2015-01-27T16:00:18.868482+00:00 heroku[web.1]: Unidling
2015-01-27T16:00:23.329359+00:00 heroku[web.1]: Starting process with command `node server.js`
2015-01-27T16:00:29.952355+00:00 app[web.1]: Create Email Queue cronjob scheduled at 0 2 0 * * * on timezone Asia/Shanghai
2015-01-27T16:00:29.962476+00:00 app[web.1]: Tue Jan 27 2015 16:00:29 GMT+0000 (UTC) | XXX app started on port 51605
2015-01-27T16:00:29.962543+00:00 app[web.1]: === LOGS ==========================================
2015-01-27T16:00:29.944710+00:00 app[web.1]: Deprecation warning: moment().zone is deprecated, use moment().utcOffset instead. https://github.com/moment/moment/issues/1779
2015-01-27T16:00:29.961339+00:00 app[web.1]: Process Email Queue cronjob scheduled at 0 */1 * * * * on timezone undefined
2015-01-27T16:00:31.005846+00:00 heroku[router]: at=info method=GET path="/WAKEUP_DYNO" host=xxx.herokuapp.com request_id=885034de-57fb-452f-ab76-dec9b6f0958d dyno=web.1 connect=6ms service=40ms status=404 bytes=245
2015-01-27T16:00:31.029943+00:00 app[scheduler.1203]: ======WAKUP DYNO: HEROKU RESPONSE: Cannot GET /WAKEUP_DYNO
2015-01-27T16:00:31.029950+00:00 app[scheduler.1203]: 
2015-01-27T16:00:31.013335+00:00 app[web.1]: GET /WAKEUP_DYNO 404 24 - 14.446 ms
2015-01-27T16:00:30.020379+00:00 heroku[web.1]: State changed from starting to up
2015-01-27T16:00:32.002333+00:00 heroku[scheduler.1203]: State changed from up to complete
2015-01-27T16:00:31.973919+00:00 heroku[scheduler.1203]: Process exited with status 0
2015-01-27T16:01:01.079431+00:00 app[web.1]: mailer.process_queue => Finished processing each email address from 0 queued emails
2015-01-27T16:01:01.078878+00:00 app[web.1]: emails: 0
2015-01-27T16:02:01.092341+00:00 app[web.1]: ====createDailyMail created daily email at: 2015-01-27 00:00:00
2015-01-27T16:02:01.086960+00:00 app[web.1]: mailer.process_queue => Finished processing each email address from 0 queued emails
2015-01-27T16:03:00.098213+00:00 app[web.1]: emails: 1
2015-01-27T16:03:00.171912+00:00 app[web.1]: mailer.process_queue sending => Email 54c7b679f952480300579948 to clarkhtse@gmail.com with subject: [XXXXXX] Stats on 2015-01-27
2015-01-27T16:03:01.216343+00:00 app[web.1]: mailer.process_queue success => Email to clarkhtse@gmail.com  Sent with response => success marked as sent
2015-01-27T16:03:01.222617+00:00 app[web.1]: mailer.process_queue => Finished processing each email address from 1 queued emails
2015-01-27T16:04:00.154263+00:00 app[web.1]: emails: 0
2015-01-27T16:04:00.154393+00:00 app[web.1]: mailer.process_queue => Finished processing each email address from 0 queued emails

There you go. Your little cronjob should work as you expect now :)

GPUImage Video Merging: Timestamp Synchronization

2015-01-20T13:19:11+00:00

One of tricky problem with GPUImage to merge multiple videos is that it could handle the frame timestamp perfectly synchronized between videos. You probably notice that the final merged output’s duration is shorter comparing to original source videos.

There are several issues on github mentioned about this already: video overlay from camera + audio #1413 and Blend two movie inputs, maintaining sync #261. As author mentioned in the comment:

The core problems come from the GPUImageTwoInputFilter, which the blends are subclassed from. Both movie sources provide their own timestamps for each frame that’s passed in, but I’m not sure that I’m keeping these timestamps straight or in sync. Timestamps are passed down the filter chain and then used for the timestamps on recording in the movie writer, but keeping them in a consistent order and based on some synchronized time value isn’t managed well.

This is what triggers all the bizarre behavior people see around blending two movies together (something I never designed things for, and now need to find a way to fix). It’s a tricky problem, and I don’t spend a lot of time working with movie files, so it’s not one of the areas I’ve put a lot of thought into.

Out-of-Sync Timestamp : Why?

Let’s recall how to setup chroma key to handle two video merging, which is quite typical use case:

NSURL *urlA = [NSURL fileURLWithPath:self.tempMoviePath];
self.gpuMovieA = [[GPUImageMovie alloc] initWithURL:urlA];

NSURL *urlB = [[NSBundle mainBundle] URLForResource:fxMovie.movieName withExtension:@"mov"];
self.gpuMovieFX = [[GPUImageMovie alloc] initWithURL:urlB];

self.filter = [[GPUImageChromaKeyBlendFilter alloc] init];
[self.filter forceProcessingAtSize:CGSizeMake(640, 640)];

[self.gpuMovieFX addTarget:self.filter];
[self.gpuMovieA addTarget:self.filter];

//setup writer
NSString *pathToMovie = [NSHomeDirectory() stringByAppendingPathComponent:@"Documents/output.mov"];
unlink([pathToMovie UTF8String]); // If a file already exists, AVAssetWriter won't let you record new frames, so delete the old movie
self.convertedVideoPath = [NSURL fileURLWithPath:pathToMovie];
self.movieWriter = [[GPUImageMovieWriter alloc] initWithMovieURL:self.convertedVideoPath size:CGSizeMake(640.0, 640.0)];
[self.filter addTarget:self.movieWriter];

[self.movieWriter startRecording];
[self.gpuMovieA startProcessing];
[self.gpuMovieFX startProcessing];

__weak typeof(self) *sself = self;
[self.movieWriter setCompletionBlock:^{
    [sself.gpuMovieFX endProcessing];
    [sself.gpuMovieA endProcessing];
    [sself.movieWriter finishRecording];
}];

Let’s add some debug print in newFrameReadyAtTime in filter and movie writer, then give it a run.

Here is the logs from console:

2015-01-20 17:40:38.870 BBB GPUImage Movie: video.mp4 duration: 8.07
2015-01-20 17:40:38.878 BBB GPUImage Movie: APP_BOOMi_Paparazzi.mp4 duration: 8.09
2015-01-20 17:40:39.035 BBB 0. processed frame at time: {1/600 = 0.002}
2015-01-20 17:40:39.074 BBB 1. processed frame at time: {1000/24000 = 0.042}
2015-01-20 17:40:39.192 BBB 0. processed frame at time: {7351/88200 = 0.083}
2015-01-20 17:40:39.207 BBB 0. processed frame at time: {11026/88200 = 0.125}
2015-01-20 17:40:39.214 BBB 0. processed frame at time: {14701/88200 = 0.167}
2015-01-20 17:40:39.227 BBB 1. processed frame at time: {2000/24000 = 0.083}
...
2015-01-20 17:40:50.245 BBB 0. processed frame at time: {697369/88200 = 7.907}
2015-01-20 17:40:50.251 BBB 1. processed frame at time: {183000/24000 = 7.625}
2015-01-20 17:40:50.305 BBB 0. processed frame at time: {701044/88200 = 7.948}
2015-01-20 17:40:50.310 BBB 1. processed frame at time: {184000/24000 = 7.667}
2015-01-20 17:40:50.366 BBB 0. processed frame at time: {704719/88200 = 7.990}
2015-01-20 17:40:50.371 BBB 1. processed frame at time: {185000/24000 = 7.708}
2015-01-20 17:40:50.425 BBB 0. processed frame at time: {708394/88200 = 8.032}
2015-01-20 17:40:50.430 BBB writer finihsed
2015-01-20 17:40:50.430 BBB 1. processed frame at time: {186000/24000 = 7.750}
2015-01-20 17:40:50.501 BBB ----FINAL OUTPUT video duration: 7.75000, audio: 0.00000, cut needed: NO

You would notice that the execution order is not follow “0, 1, 0, 1” strictly, yeah, that’s the problem, which causes, in the end, the last frame got written is 7.75000 rather than 8.000.

The reason lies in the readNextVideoFrameFromOutput in GPUImageMovie, where it handles reading and processing frame:

- (BOOL)readNextVideoFrameFromOutput:(AVAssetReaderOutput *)readerVideoTrackOutput;
{
    if (reader.status == AVAssetReaderStatusReading && ! videoEncodingIsFinished)
    {
        CMSampleBufferRef sampleBufferRef = [readerVideoTrackOutput copyNextSampleBuffer];
        if (sampleBufferRef) 
        {
            //... other codes
            __unsafe_unretained GPUImageMovie *weakSelf = self;
            runSynchronouslyOnVideoProcessingQueue(^{
                [weakSelf processMovieFrame:sampleBufferRef];
                CMSampleBufferInvalidate(sampleBufferRef);
                CFRelease(sampleBufferRef);
            });
            return YES;
        }
}

This line runSynchronouslyOnVideoProcessingQueue is the key to the answer. It simply dispatch the processing frame block to a serial queue across all movies. But there is no dispatch-order-management, which causes a race competition among all movies when all of them got sample buffers ready to be processed.

Because it is a serial queue, so the execution order is one at a time, hence fixed, but the dispatch block order is kinda random.

We need take a new approach to solve the dispatch order to make sure it follow strictly 01010101 order.

Dispatch Block Order To Save

The idea is to have each movie has two extra properties: a boolean pauseVideoRead and a block onProcessMovieFrameDone.

The boolean pauseVideoRead is used to pause other movies’s asset reading when one movie is being processed.

The block onProcessMovieFrameDone is called at the end of frame processing and notify outside to pause current movie asset reading and resume asset reading and processing of next movie in pre-defined execution order list.

First, go to GPUImageMovie and change method readNextVideoFrameFromOutput:

- (BOOL)readNextVideoFrameFromOutput:(AVAssetReaderOutput *)readerVideoTrackOutput;
{
    if (reader.status == AVAssetReaderStatusReading && ! videoEncodingIsFinished && !self.pauseVideoRead)
    {
        CMSampleBufferRef sampleBufferRef = [readerVideoTrackOutput copyNextSampleBuffer];
        if (sampleBufferRef) 
        {
            //...
            __unsafe_unretained GPUImageMovie *weakSelf = self;
            runSynchronouslyOnVideoProcessingQueue(^{
                [weakSelf processMovieFrame:sampleBufferRef];
                CMTime currentSampleTime = CMSampleBufferGetOutputPresentationTimeStamp(sampleBufferRef);
                CMSampleBufferInvalidate(sampleBufferRef);
                CFRelease(sampleBufferRef);
                
                if(weakSelf.onProcessMovieFrameDone){
                    weakSelf.onProcessMovieFrameDone(weakSelf, currentSampleTime);
                }
            });

            return YES;
        }
    }
}

Quite a simple change and let’s look the demo code in chroma key.

NSURL *urlA = [NSURL fileURLWithPath:self.tempMoviePath];
self.gpuMovieA = [[GPUImageMovie alloc] initWithURL:urlA];

NSURL *urlB = [[NSBundle mainBundle] URLForResource:fxMovie.movieName withExtension:@"mov"];
self.gpuMovieFX = [[GPUImageMovie alloc] initWithURL:urlB];

self.filter = [[GPUImageChromaKeyBlendFilter alloc] init];
[self.filter forceProcessingAtSize:CGSizeMake(640, 640)];

[self.gpuMovieFX addTarget:self.filter];
[self.gpuMovieA addTarget:self.filter];

//setup writer
NSString *pathToMovie = [NSHomeDirectory() stringByAppendingPathComponent:@"Documents/output.mov"];
unlink([pathToMovie UTF8String]); // If a file already exists, AVAssetWriter won't let you record new frames, so delete the old movie
self.convertedVideoPath = [NSURL fileURLWithPath:pathToMovie];
self.movieWriter = [[GPUImageMovieWriter alloc] initWithMovieURL:self.convertedVideoPath size:CGSizeMake(640.0, 640.0)];
[self.filter addTarget:self.movieWriter];

NSMutableArray *allMoviesByExecutionOrder = [NSMutableArray array];
[allMoviesByExecutionOrder addObject: self.gpuMovieA];
[allMoviesByExecutionOrder addObject: self.gpuMovieFX];

for(NSInteger i=0; i < allMoviesByExecutionOrder.count; i++){
    GPUImageMovie *movie = allMoviesByExecutionOrder[i];
    movie.pauseVideoRead = i != 0; //make first movie default to be not paused
}

//process each movie by orders in allMoviesByExecutionOrder
void (^onProcessFrameBlock)(GPUImageMovie *, CMTime) = ^(GPUImageMovie *movie, CMTime frameTime){
    movie.pauseVideoRead = YES;
    NSUInteger index = [allMoviesByExecutionOrder indexOfObject:movie];
    //NSLog(@"%d.Movie: %@ processed frame at time: %@", index,movie.url.lastPathComponent,CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, frameTime)));
    if(index != NSNotFound){
        if(index == allMoviesByExecutionOrder.count - 1){
            index = 0;
        }else{
            index++;
        }
        GPUImageMovie *nextMovie = allMoviesByExecutionOrder[index];
        nextMovie.pauseVideoRead = NO;
    }
};

for (GPUImageMovie *movie in allMoviesByExecutionOrder) {
    movie.onProcessMovieFrameDone = onProcessFrameBlock;
}

//ready to go
[self.movieWriter startRecording];
for (GPUImageMovie *movie in allMoviesByExecutionOrder) {
    [movie startProcessing];
}

__weak typeof(self) *sself = self;
[self.movieWriter setCompletionBlock:^{
    for (GPUImageMovie *movie in allMoviesByExecutionOrder) {
        [movie endProcessing];
    }
    [sself.movieWriter finishRecording];
}];

allMoviesByExecutionOrder is very important to be noticed. It defines the order of movies to dispatch to be processed. And inside onProcessFrameBlock, we pause current movie’s reading and resume next movie’s reading, so that it would ensure all movies are dispatched with correct order to prevent race competition.

Following is the part of logs from console after that:

2015-01-20 17:36:39.936 BBB GPUImage Movie: video.mp4 duration: 8.07
2015-01-20 17:36:39.942 BBB GPUImage Movie: APP_BOOMi_Paparazzi.mp4 duration: 8.09
2015-01-20 17:36:40.138 BBB 0. processed frame at time: {1/600 = 0.002}
2015-01-20 17:36:40.151 BBB 1. processed frame at time: {1000/24000 = 0.042}
2015-01-20 17:36:40.296 BBB 0. processed frame at time: {7351/88200 = 0.083}
2015-01-20 17:36:40.311 BBB 1. processed frame at time: {2000/24000 = 0.083}
2015-01-20 17:36:40.369 BBB 0. processed frame at time: {11026/88200 = 0.125}
...
2015-01-20 17:36:52.340 BBB 0. processed frame at time: {697369/88200 = 7.907}
2015-01-20 17:36:52.346 BBB 1. processed frame at time: {190000/24000 = 7.917}
2015-01-20 17:36:52.402 BBB 0. processed frame at time: {701044/88200 = 7.948}
2015-01-20 17:36:52.410 BBB 1. processed frame at time: {191000/24000 = 7.958}
2015-01-20 17:36:52.465 BBB 0. processed frame at time: {704719/88200 = 7.990}
2015-01-20 17:36:52.476 BBB 1. processed frame at time: {192000/24000 = 8.000}
2015-01-20 17:36:52.530 BBB 0. processed frame at time: {708394/88200 = 8.032}
2015-01-20 17:36:52.535 BBB writer finihsed
2015-01-20 17:36:52.550 BBB ----FINAL OUTPUT video duration: 8.00000, audio: 0.00000, cut needed: NO

Now you see the running order is strictly one after another, and final output shows the duration is perfect, yay!

Performance

You may wonder since we restrict the order, would it have an impact on performance?

Here is the stats on iPod5 running multiple videos(8 seconds, 640*640 resolution) merging:

Two Videos - before 10 seconds, after 11 seconds
Three Videos - before 11 seconds, after 12 seconds
Four Videos - before 12 seconds, after 16 seconds

	Before(Order Dynamic)	After(Order fixed)
Two Videos	10s	11.0s
Three Videos	11s	12.00s
Four Videos	12s	15.96s

Only when you process four video merging, you probably would notice the difference; for other cases, the difference is just trivial.

All changes are available on github also: Fix Video Timestamp Inconsistence - which video stops too early.

GPUImage Movie Writer: Merging All Audio Tracks From Multiple Movies

2015-01-11T14:13:24+00:00

One tough problem when using GPUImage movie write output video with audio is that it only support one audio track. There is a github issue on it: GPUImageMovieWriter can’t export audiomix audio tracks issue #1834, Audio writing issues with GPUImageMovieWriter and Merging of video and audio.

The author of GPUImage mentioned that in issue Merging of video and audio #1223:

There currently is no way to do this kind of audio mixing. Only one audio source is used at a time.

But why GPUImageMovieWriter only support one audio track writing?!!! This really limits its capability and it shouldn’t be like that.

![jackie_chan_by_rober_raik-d4cly01](

#GPUImageMovieWriter: One Audio Source Only?

The reason that cause it only could handle one audio source during writing lies in the structure.

First, let’s recall the code to do audio writing setup:

NSURL *urlA = [NSURL fileURLWithPath:self.tempMoviePath];
self.gpuMovieA = [[GPUImageMovie alloc] initWithURL:urlA];

NSURL *urlB = [[NSBundle mainBundle] URLForResource:fxMovie.movieName withExtension:@"mov"];
self.gpuMovieFX = [[GPUImageMovie alloc] initWithURL:urlB];

self.filter = [[GPUImageChromaKeyBlendFilter alloc] init];
[self.filter forceProcessingAtSize:CGSizeMake(640, 640)];

[self.gpuMovieFX addTarget:self.filter];
[self.gpuMovieA addTarget:self.filter];

//setup writer
NSString *pathToMovie = [NSHomeDirectory() stringByAppendingPathComponent:
        [NSString stringWithFormat:@"Documents/output.mov"];
unlink([pathToMovie UTF8String]); // If a file already exists, AVAssetWriter won't let you record new frames, so delete the old movie
self.convertedVideoPath = [NSURL fileURLWithPath:pathToMovie];
self.movieWriter = [[GPUImageMovieWriter alloc] initWithMovieURL:self.convertedVideoPath
                                                            size:CGSizeMake(640.0, 640.0)];
[self.filter addTarget:self.movieWriter];    

self.movieWriter.shouldPassthroughAudio = YES;
self.gpuMovieA.audioEncodingTarget = self.movieWriter;
//add this line otherwise it will cause "Had to drop an audio frame" which cozes the saved video lose some sounds
[self.gpuMovieA enableSynchronizedEncodingUsingMovieWriter:self.movieWriter];

[self.movieWriter startRecording];
[self.gpuMovieA startProcessing];
[self.gpuMovieFX startProcessing];

Yes, it is this code who does the audio writing setup:

self.movieWriter.shouldPassthroughAudio = YES;
self.gpuMovieA.audioEncodingTarget = self.movieWriter;
//add this line otherwise it will cause "Had to drop an audio frame" which cozes the saved video lose some sounds
[self.gpuMovieA enableSynchronizedEncodingUsingMovieWriter:self.movieWriter];

Here you tell writer saying “hey, I want you to writer audio from gpuMovieA to final output video”.

Once you dive into the code enableSynchronizedEncodingUsingMovieWriter in GPUImageMovie, it has called important method readNextAudioSampleFromOutput,which is kinda like following:

- (BOOL)readNextAudioSampleFromOutput:(AVAssetReaderOutput *)readerAudioTrackOutput;
{
    if (reader.status == AVAssetReaderStatusReading && ! audioEncodingIsFinished)
    {
        CMSampleBufferRef audioSampleBufferRef = [readerAudioTrackOutput copyNextSampleBuffer];
        if (audioSampleBufferRef)
        {
            //NSLog(@"read an audio frame: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, CMSampleBufferGetOutputPresentationTimeStamp(audioSampleBufferRef))));
            [self.audioEncodingTarget processAudioBuffer:audioSampleBufferRef];
            CFRelease(audioSampleBufferRef);
            return YES;
        }
        ...
        }
    ...
    }
}

Now you understand why it could only handle one audio writing. It is because audio’s sample buffer is not like video’s sample buffer which you could upload to GPU, render it using shader, then pull it down and write to disk. AKA: there is no way you could merge two audio sample buffer to create a “combining” effect like video.

We need take a new approach.

Using AVComposition to Mix Multiple Audio Tracks

There you go, the new approach is to use AVComposition to merge two or more audio tracks together. I have written a blog on this already: GPUImage Merge Videos with Chroma Key - GPUImageMovieWriter Two Audio Tracks, but that is just a very very raw spike, it doesn’t synced with video writing, which is not acceptable.

GPUImageMovie

First, let’s see how we modified the GPUImageMovie by simply adding one method (void)loadAsset:(dispatch_group_t)readyGroup; and change other little bit:

- (void)loadAsset:(dispatch_group_t)readyGroup {
    NSAssert(self.url || self.asset, @"Url or asset should be passed for initial");
    NSAssert(readyGroup, @"should pass a ready group to tell when i'm ready");
    dispatch_group_enter(readyGroup);

    /*
    if( self.playerItem ) {
        [self processPlayerItem];
        return;
    }
    */
    if(self.url == nil && self.asset != nil)
    {
        //directly inited from outside
        dispatch_group_leave(readyGroup);
        return;
    }
    if (_shouldRepeat) keepLooping = YES;

    previousFrameTime = kCMTimeZero;
    previousActualFrameTime = CFAbsoluteTimeGetCurrent();

    NSDictionary *inputOptions = [NSDictionary dictionaryWithObject:[NSNumber numberWithBool:YES] forKey:AVURLAssetPreferPreciseDurationAndTimingKey];
    AVURLAsset *inputAsset = [[AVURLAsset alloc] initWithURL:self.url options:inputOptions];

    GPUImageMovie __block *blockSelf = self;

    [inputAsset loadValuesAsynchronouslyForKeys:[NSArray arrayWithObject:@"tracks"] completionHandler: ^{
        dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
            NSError *error = nil;
            AVKeyValueStatus tracksStatus = [inputAsset statusOfValueForKey:@"tracks" error:&error];
            if (tracksStatus != AVKeyValueStatusLoaded)
            {
                return;
            }
            NSLog(@"GPUImageMovie %@ loaded successfully", blockSelf.url.lastPathComponent);
            blockSelf.asset = inputAsset;
            dispatch_group_leave(readyGroup);
        });
    }];

}

- (void)startProcessing
{
    dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
        if( self.playerItem ) {
            [self processPlayerItem];
            return;
        }
        [self processAsset];
    });
}

So basically we moved out the asset initialization code to loadAsset and now in startProcessing it only need to kick off processAsset(asset reading).

You may wonder what’s the parameter in loadAsset is doing. Yeah, it is quite simple as GPUImageMovie need load tracks async for asset, we need a way to notify others that my asset is fully loaded and ready and I’m really good to kick asset reading part.

GPUImageMovieWriter

Let’s move to GPUImageMovieWriter class.

This is gonna be more change comparing to GPUImageMovie, but it is quite simple. We start by adding two variables:

@interface GPUImageMovieWriter : NSObject <GPUImageInput>
{
    BOOL alreadyFinishedRecording;
    ...
    AVAssetWriterInputPixelBufferAdaptor *assetWriterPixelBufferInput;

    //This is two new varaibles
    AVAssetReader *assetAudioReader;
    AVAssetReaderAudioMixOutput *assetAudioReaderTrackOutput;

Followed by adding setupAudioReaderWithTracks method to setup audio reader that mixing multiple audio tracks from movie:

- (void)setupAudioReaderWithTracks:(NSMutableArray *)audioTracks {
    if(audioTracks.count > 0){
        AVMutableComposition* mixComposition = [AVMutableComposition composition];

        for(AVAssetTrack *track in audioTracks){

            NSLog(@"track url: %@ duration: %.2f", track.asset, CMTimeGetSeconds(track.asset.duration));
            AVMutableCompositionTrack *compositionCommentaryTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeAudio

                                                                                                preferredTrackID:kCMPersistentTrackID_Invalid];
            [compositionCommentaryTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, track.asset.duration)
                                                ofTrack:track
                                                 atTime:kCMTimeZero error:nil];

        }

        assetAudioReader = [AVAssetReader assetReaderWithAsset:mixComposition error:nil];
        assetAudioReaderTrackOutput =
                [[AVAssetReaderAudioMixOutput alloc] initWithAudioTracks:[mixComposition tracksWithMediaType:AVMediaTypeAudio]
                                                           audioSettings:nil];

        [assetAudioReader addOutput:assetAudioReaderTrackOutput];
    }
}

Nothing fancy except AVAssetReaderAudioMixOutput this part, which is magic :)

Then - (void)startAudioRecording to start reading, so that our audio reader is ready to go!

- (void)startAudioRecording {
    if(assetAudioReader){
        if(![assetAudioReader startReading]){
            NSLog(@"asset audio reader start reading failed: %@", assetAudioReader.error);
        }
    }
}

Next is the audio writing part, - (void)startAudioWritingWithComplectionBlock:(void (^)())completionBlock:

- (void)startAudioWritingWithComplectionBlock:(void (^)())completionBlock {
    if(assetAudioReader){
        audioQueue = dispatch_queue_create("com.sunsetlakesoftware.GPUImage.audioReadingQueue", NULL);
        [assetWriterAudioInput requestMediaDataWhenReadyOnQueue:audioQueue usingBlock:^{
            // Because the block is called asynchronously, check to see whether its task is complete.
            BOOL completedOrFailed = NO;
            // If the task isn't complete yet, make sure that the input is actually ready for more media data.
            while ([assetWriterAudioInput isReadyForMoreMediaData] && !completedOrFailed) {
                // Get the next audio sample buffer, and append it to the output file.
                CMSampleBufferRef sampleBuffer = [assetAudioReaderTrackOutput copyNextSampleBuffer];
                if (sampleBuffer != NULL) {
                    BOOL success = [assetWriterAudioInput appendSampleBuffer:sampleBuffer];
                    CFRelease(sampleBuffer);
                    sampleBuffer = NULL;
                    completedOrFailed = !success;
                }
                else {
                    completedOrFailed = YES;
                }
            }//end of loop

            if (completedOrFailed) {
                NSLog(@"audio wrint done");
                // Mark the input as finished, but only if we haven't already done so, and then leave the dispatch group (since the audio work has finished).
                audioEncodingIsFinished = YES;
                [assetWriterAudioInput markAsFinished];
                if(completionBlock){
                    completionBlock();
                }
            }
        }];
    } else{
        NSLog(@" no audio reader is being set, this could happen when no audio tracks being set");
        if(completionBlock){
            completionBlock();
        }
    }
}

Again, nothing quite special here. One thing to note is that we dont’ call asset write finishing writing when audio wrting is done because that we need check video writing part, which means that we also need to add new method for handling video wrting finish.

- (void)finishVideoRecordingWithCompletionHandler:(void (^)(void))handler{

    runSynchronouslyOnContextQueue(_movieWriterContext, ^{
        isRecording = NO;
        videoEncodingIsFinished = YES;
        [assetWriterVideoInput markAsFinished];
        
        //ATTENTION: DO NOT CALL [assetWrite finishWriting] HERE

        if (handler)
            runAsynchronouslyOnContextQueue(_movieWriterContext, handler);
    });

}

The last thing which is also very important is that we need to modify - (void)startRecording to uncomment last line.

- (void)startRecording;
{
    alreadyFinishedRecording = NO;
    startTime = kCMTimeInvalid;
    runSynchronouslyOnContextQueue(_movieWriterContext, ^{
        if (audioInputReadyCallback == NULL)
        {
            [assetWriter startWriting];
        }
    });
    isRecording = YES;
    [assetWriter startSessionAtSourceTime:kCMTimeZero];
}

[assetWriter startSessionAtSourceTime:kCMTimeZero]; this line of code has to be fired!

That’s it for GPUImageMovieWriter. And That’s all the changes for GPUImage part, but how the demo running code is gonna be looked like ?

Demo Time

We’re all set, let’s take a look the demo code:

self.gpuMovieA = [[GPUImageMovie alloc] initWithURL:rawVideoURL];    
self.gpuMovieFX = [[GPUImageMovie alloc] initWithURL:fxURL];
    
self.filter = [[GPUImageChromaKeyBlendFilter alloc] init];
[self.gpuMovieFX addTarget:self.filter];
[self.gpuMovieA addTarget:self.filter];

//setup writer
NSString *pathToMovie = [NSHomeDirectory() stringByAppendingPathComponent:@"Documents/gpu_output.mov"];
unlink([pathToMovie UTF8String]); // If a file already exists, AVAssetWriter won't let you record new frames, so delete the old movie
self.outputURL = [NSURL fileURLWithPath:pathToMovie];
self.movieWriter =  [[GPUImageMovieWriter alloc] initWithMovieURL:self.outputURL size:CGSizeMake(640.0/2, 640.0/2)];
[self.filter addTarget:self.movieWriter];

NSArray *movies = @[self.gpuMovieA, self.gpuMovieFX];
dispatch_group_t movieReadyDispatchGroup = dispatch_group_create();
for(GPUImageMovie *movie in movies){
    [movie loadAsset:movieReadyDispatchGroup];
}

__weak typeof(self) weakSelf = self;
dispatch_group_notify(movieReadyDispatchGroup, dispatch_get_main_queue(), ^{
    NSLog(@"all movies are ready to process :)");
    NSMutableArray *audioTracks = [NSMutableArray array];
    for(GPUImageMovie *movie in movies){
        AVAssetTrack *track = [movie.asset tracksWithMediaType:AVMediaTypeAudio].firstObject;
        if(track){
            [audioTracks addObject:track];
        }
    }
    
    if(audioTracks.count > 0){
        [self.movieWriter setupAudioReaderWithTracks:audioTracks];
        [self.movieWriter setHasAudioTrack:YES]; //use default audio settings, setup asset writer audio
    }

    self.recordSyncingDispatchGroup = dispatch_group_create();

    //this has to be called before, to make sure all audio/video in movie writer is set
    [self.movieWriter startRecording];
    [self.movieWriter startAudioRecording];

    //video handling
    dispatch_group_enter(self.recordSyncingDispatchGroup);
    [self.gpuMovieA startProcessing];
    [self.gpuMovieFX startProcessing];
    [self.movieWriter setCompletionBlock:^{
        [weakSelf.gpuMovieFX endProcessing];
        [weakSelf.gpuMovieA endProcessing];
        [weakSelf.movieWriter finishVideoRecordingWithCompletionHandler:^{
            NSLog(@"===video wrote is done");
            dispatch_group_leave(weakSelf.recordSyncingDispatchGroup);
        }];
    }];
    
    //audio handling
    dispatch_group_enter(self.recordSyncingDispatchGroup);
    [self.movieWriter startAudioRecording];
    [self.movieWriter startAudioWritingWithComplectionBlock:^{
        NSLog(@"====audio wring is done");
        dispatch_group_leave(weakSelf.recordSyncingDispatchGroup);
    }];
    
    dispatch_group_notify(self.recordSyncingDispatchGroup, dispatch_get_main_queue(), ^{
        NSLog(@"vidoe and audio writing are both done-----------------");
        [self.movieWriter finishRecordingWithCompletionHandler:^{
            NSLog(@"final clean up is done :)");
        }];
    });

});

Let’s start with this part:

NSArray *movies = @[self.gpuMovieA, self.gpuMovieFX];
dispatch_group_t movieReadyDispatchGroup = dispatch_group_create();
for(GPUImageMovie *movie in movies){
    [movie loadAsset:movieReadyDispatchGroup];
}
__weak typeof(self) weakSelf = self;
dispatch_group_notify(movieReadyDispatchGroup, dispatch_get_main_queue(), ^{
    NSLog(@"all movies are ready to process :)");
    NSMutableArray *audioTracks = [NSMutableArray array];
    for(GPUImageMovie *movie in movies){
        AVAssetTrack *track = [movie.asset tracksWithMediaType:AVMediaTypeAudio].firstObject;
        if(track){
            [audioTracks addObject:track];
        }
    }
}

We have used a dispatch group to make sure all GPUImageMovies has finished their initialization process, which at that time all assets are ready. Then we got all the audio tracks from movies for later use.

if(audioTracks.count > 0){
    [self.movieWriter setupAudioReaderWithTracks:audioTracks]; //setup audio readers from all tracks
    [self.movieWriter setHasAudioTrack:YES]; //use default audio settings, setup asset writer audio
}

This part is really easy, we check if any track is available, then tell movie writer to setup audio reader and writer.

self.recordSyncingDispatchGroup = dispatch_group_create();

//this has to be called before, to make sure all audio/video in movie writer is set
[self.movieWriter startRecording];

Here we use another dispatch group who helps us to cooridinate audio and video writing part as those two are both asynchrously. Then we fire startRecording,which just starts asset writer.

//video handling
dispatch_group_enter(self.recordSyncingDispatchGroup);
[self.gpuMovieA startProcessing];
[self.gpuMovieFX startProcessing];
[self.movieWriter setCompletionBlock:^{
    [weakSelf.gpuMovieFX endProcessing];
    [weakSelf.gpuMovieA endProcessing];
    [weakSelf.movieWriter finishVideoRecordingWithCompletionHandler:^{
        NSLog(@"===video wrote is done");
        dispatch_group_leave(weakSelf.recordSyncingDispatchGroup);
    }];
}];

This is the video writing part, be careful with finishVideoRecordingWithCompletionHandler as it is totally wrong if you call original method finishRecordingWithCompletionHandler or finishRecording because we control how movie writer when and how it should be finished.

//audio handling
dispatch_group_enter(self.recordSyncingDispatchGroup);
[self.movieWriter startAudioRecording];
[self.movieWriter startAudioWritingWithComplectionBlock:^{
    NSLog(@"====audio wring is done");
    dispatch_group_leave(weakSelf.recordSyncingDispatchGroup);
}];

Audio handling is pretty much same like video.(ps: I really should rename those methods name to be more consistent. For example, startAudioRecording acutally should be renamed to startAudioReading, and startAudioWritingWithComplectionBlock should be finishAudioWritingWithComplectionBlock)

Last part is the final completion code that video and audio are both finished their writing:

dispatch_group_notify(self.recordSyncingDispatchGroup, dispatch_get_main_queue(), ^{
    NSLog(@"vidoe and audio writing are both done-----------------");
    [self.movieWriter finishRecordingWithCompletionHandler:^{
        NSLog(@"final clean up is done :)");
        //you could fire some UI work on main queue
    }];
})

That’s it, give a shoot in XCode, you should be able to what you expected! :)

Conclusion

Man, working with low level threading and avfoundation is a challenage, working on a very complex existing codebase is damn super ultra challenge.

I help someone could give some suggestions to make this library even better :)

All code and examples(which covers all cases: no audio track, one audio track, all audio tracks) are available on github: GPUImageMultpileAudioTracksMerge

And GPUImage changes are available on github also: added support for merging all audio tracks from multiple movies, you should be easy to get this pull request.

Video Merging with AVFoundation and OpenGL ES on iOS: Optimization With Instruments

2015-01-10T04:25:44+00:00

Now we have our pretty much all the function working, next we need do some performance optimization.

But first, let’s do some testing. By change video from 640x640 to 1280x720, the reason to choose 1280x720 is the app that we mentioned before Action Movie FX, is using this resolution for video that it got from camera. Other videos like FX and Alpha we’re gonna use 960x720 which is high resolution video the app is gonna use. All movies are available on github. All movies’s duraton is about 5.52 seconds.

Before we did any optimization, our project keeps throwing “Low memory warning”, and sometimes even crashes. If you wonder how GPUImage performance, well, it just crashes immediately. Obviously this is not acceptable.

After optimzation we’re gonna do in this article, our project works quite well, no crashes and we got some comparison result:

ipod5 - 5.7 seconds, action movie fx - 10.6 seconds
iPhone4s - 6.7s ~ 7.380 seconds , action movie fx - 17.28 seconds
iPhone4 - 12.0s~13.0s seconds , action movie fx - 36 seconds

	AVFoundataion + OpenGL ES	Action Movie FX
iPhone4	12.0s ~ 13.0s	36.1s
iPhone4S	6.7s ~ 7.38s	17.28s
iPod5	5.7s	10.6s

As you see, it is actually quite big difference out there!

The complete code is on the github: ThreeVidoes-Final. You’re free to download and try it.

OpenGL ES Analyzer

Apple has a great documentation covered this topic: Tuning Your OpenGL ES App.

Open XCode, try “Product -> Profile”, select OpenGL ES Analyzer in instruments dialog, then click run button in top left part of instrument.

Here is the result we got:

We could list all those points:

Redudant Call
Recommend Using VBO
Unitialized Texture Data
Logical Buffer Store
CPU wait on GPU for Finish
Draw Call Accessed Vertex Attributes
Recommend Using VAO
Logic Buffer Load

From my knowledge, I will start with Recommend Using VBO and Recommend Using VAO first, as it is easier to start with. Another reason is that other points like Unitialized Texture Data and CPU wait on GPU for Finish are not quite important or deliberately left like that. Well, Logic Buffer Load and Logical Buffer Store to be honest, I’m not quite sure what they are about and I hope someone could share some light on it :)

In VideoWriter, here is the current implementation in kickoffRecording method:

glViewport(0, 0, (int)self.videoSize.width, (int)self.videoSize.height);
//use shader program
NSAssert(_program, @"Program should be created");
glUseProgram(_program);


// This needs to be flipped to write out to video correctly
static const GLfloat squareVertices[] = {
        -1.0f, -1.0f,
        1.0f, -1.0f,
        -1.0f,  1.0f,
        1.0f,  1.0f,
};

static const GLfloat textureCoordinates[] = {
        0.0f, 0.0f,
        1.0f, 0.0f,
        0.0f, 1.0f,
        1.0f, 1.0f,
};
//https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/TechniquesforWorkingwithVertexData/TechniquesforWorkingwithVertexData.html
glVertexAttribPointer(_positionSlot, 2, GL_FLOAT, 0, 0, squareVertices);
glVertexAttribPointer(_srcTexCoord1Slot, 2, GL_FLOAT, 0, 0, textureCoordinates);
glVertexAttribPointer(_srcTexCoord2Slot, 2, GL_FLOAT, 0, 0, textureCoordinates);
glVertexAttribPointer(_srcTexCoord3Slot, 2, GL_FLOAT, 0, 0, textureCoordinates);

//bind uniforms
...
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glFinish();

Well, how do we use VBO? What’s VBO?

VBO, according to Vertex Buffer Objects:

A Vertex Buffer Object (VBO) is an OpenGL feature that provides methods for uploading vertex data (position, normal vector, color, etc.) to the video device for non-immediate-mode rendering. VBOs offer substantial performance gains over immediate mode rendering primarily because the data resides in the video device memory rather than the system memory and so it can be rendered directly by the video device.

Also, Apple has a great article on it: Best Practices for Working with Vertex Data, which also covers VBO and VAO that we’gonna talk later.

If you feel still it is little bit hard to understand it, and it would be great it has some demo/example code along it, RayWenderlich has a greate blog on it: OpenGL Tutorial for iOS: OpenGL ES 2.0.

Okay, let’s start coding by declaring some constants and struts:

typedef struct {
    GLfloat position[2];
    GLfloat texcoord[2];
} Vertex;

const Vertex Vertices[] = {
        { { -1, -1 }, { 0, 0}}, //bottom left
        { {1, -1 }, {1, 0}},  //bottom right
        { {-1, 1 }, {0, 1}}, //top left
        { {1, 1 }, {1, 1}}   //top right
};

const GLubyte Indices[] = {
        0, 1, 2,
        2, 1, 3
};

Then we create setupVBO method in initial setup code:

- (void)setupVBOs {
    glGenBuffers(1, &vertexBuffer);
    glBindBuffer(GL_ARRAY_BUFFER, vertexBuffer);
    glBufferData(GL_ARRAY_BUFFER, sizeof(Vertices), Vertices, GL_STATIC_DRAW);
    glBindBuffer(GL_ARRAY_BUFFER, 0);

    glGenBuffers(1, &indexBuffer);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexBuffer);
    glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(Indices), Indices, GL_STATIC_DRAW);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, 0);
}

Finally in kick off video writing method:

glViewport(0, 0, (int)self.videoSize.width, (int)self.videoSize.height);
//use shader program
NSAssert(_program, @"Program should be created");
glUseProgram(_program);


glBindBuffer(GL_ARRAY_BUFFER, vertexBuffer);
glVertexAttribPointer(_positionSlot, 2, GL_FLOAT, GL_FALSE,
        sizeof(Vertex), 0);

glVertexAttribPointer(_srcTexCoord1Slot, 2, GL_FLOAT, GL_FALSE,
        sizeof(Vertex), (GLvoid*) (sizeof(float) * 2));
glVertexAttribPointer(_srcTexCoord2Slot, 2, GL_FLOAT, GL_FALSE,
        sizeof(Vertex), (GLvoid*) (sizeof(float) * 2));
glVertexAttribPointer(_srcTexCoord3Slot, 2, GL_FLOAT, GL_FALSE,
        sizeof(Vertex), (GLvoid*) (sizeof(float) * 2));

//bind uniforms
...

glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexBuffer);
glDrawElements(GL_TRIANGLE_STRIP, sizeof(Indices)/sizeof(GLubyte), GL_UNSIGNED_BYTE, 0);

That’s all for using VBO to improve the vertex performance. Nothing fancy.

Run the code to make sure everything still works.

Cool, we already implemented the vertex buffer objects, but that’s not the end. We could keep going by introducing VAO(Vertex Array Objects).

Remove the setupVBO method and add a new method setupVAO:

//setup vertex array object for better performance as the position/texcoordiates are same
- (void)setupVAO {

    //create and bind a vao
    glGenVertexArraysOES(1, &vertextArrayObject);
    glBindVertexArrayOES(vertextArrayObject);

    //create and bind a BO for vertex data
    glGenBuffers(1, &vertexBuffer);
    glBindBuffer(GL_ARRAY_BUFFER, vertexBuffer);

    // copy data into the buffer object
    glBufferData(GL_ARRAY_BUFFER, sizeof(Vertices), Vertices, GL_STATIC_DRAW);

    // set up vertex attributes
    glEnableVertexAttribArray(_positionSlot);
    glVertexAttribPointer(_positionSlot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, position));
    glEnableVertexAttribArray(_srcTexCoord1Slot);
    glVertexAttribPointer(_srcTexCoord1Slot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, texcoord));


    // Create and bind a BO for index data
    glGenBuffers(1, &indexBuffer);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexBuffer);

    // copy data into the buffer object
    glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(Indices), Indices, GL_STATIC_DRAW);

    //unbind it, rebind it only when you needed
    glBindVertexArrayOES(0);
}

Then we need to slightly modify the code in rendering:

glViewport(0, 0, (int) self.videoSize.width, (int) self.videoSize.height);
//use shader program
NSAssert(_program, @"Program should be created");
glUseProgram(_program);
glBindFramebuffer(GL_FRAMEBUFFER, _frameBuffer);

// This is it. Binding the VAO again restores all buffer
// bindings and attribute settings that were previously set up
glBindVertexArrayOES(vertextArrayObject);

//Bind uniforms
...

glDrawElements(GL_TRIANGLES, sizeof(Indices) / sizeof(GLubyte), GL_UNSIGNED_BYTE, (void *) 0);
glFinish();

There you go. Run the code to see any differences. Now you have finished vertex optimization, and no more warning in instruments and it got much faster.

#Conclusion

This is the last article in this series.

I have done other cleanup also. But for other points it listed from instruments, I dont’ quite understand how to improve it. I help someone could share some experience on it and make it better.

Beside that, you could use kCVPixelFormatType_420YpCbCr8BiPlanarFullRange over kCVPixelFormatType_32BGRA for better performance, since I dont’ have much time for it, I’m just gonna leave it for someone who is interested on this :)

Video Merging with AVFoundation and OpenGL ES on iOS: Multiple Audio Tracks Merge

2015-01-10T03:01:48+00:00

One of problem with GPUImage library is that it couldn’t write multiple audio tracks, you’re only able to write one audio track at one time. But for our spike, we’re gonna solve this problem by adopting a different perspective.

To merge multiple audio tracks from movies, we need to use AVMutableComposition which allows to create composition from different assets, i.e. here is the audio tracks. Then we use AVAssetReaderAudioMixOutput to read audio samples that result from mixing the audio from one or more tracks. Then we use dispatch_group to coordinate video and audio writing process.

The complete code is on the github: ThreeVidoes-Final. You’re free to download and try it.

Prepare Audio Tracks

In VideoReader class, we add method to get audio tracks:

- (AVAssetTrack *)audioTrack{
    NSAssert(self.asset, @"Asset should be inited before access");
    NSArray *audioTracks = [self.asset tracksWithMediaType:AVMediaTypeAudio];
    return audioTracks.firstObject;
}

Setup Audio AssetReader

Then in VideoWriter, we need to setup audio asset reader from avcomposition:

- (void)setupAssetAudioReaderAndWriter {
    NSArray *audioTracks = [@[self.readerFX, self.readerRaw, self.readerAlpha] valueForKey:@"audioTrack"];
    NSLog(@"audioTracks: %@", audioTracks);

    AVMutableComposition* mixComposition = [AVMutableComposition composition];

    for(AVAssetTrack *track in audioTracks){
        if(![track isKindOfClass:[NSNull class]]){
            NSLog(@"track url: %@ duration: %.2f", track.asset, CMTimeGetSeconds(track.asset.duration));
            AVMutableCompositionTrack *compositionCommentaryTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeAudio

                                                                                                preferredTrackID:kCMPersistentTrackID_Invalid];
            [compositionCommentaryTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, track.asset.duration)
                                                ofTrack:track
                                                 atTime:kCMTimeZero error:nil];
        }
    }

    self.assetAudioReader = [AVAssetReader assetReaderWithAsset:mixComposition error:nil];
    self.assetAudioReaderTrackOutput =
            [[AVAssetReaderAudioMixOutput alloc] initWithAudioTracks:[mixComposition tracksWithMediaType:AVMediaTypeAudio]
                                                       audioSettings:nil];

    [self.assetAudioReader addOutput:self.assetAudioReaderTrackOutput];

    NSAssert(self.assetWriter, @"Writer should be inited");

    //Use default audio outputsettings

    //http://stackoverflow.com/questions/4149963/this-code-to-write-videoaudio-through-avassetwriter-and-avassetwriterinputs-is
    // Add the audio input
    AudioChannelLayout acl;
    bzero( &acl, sizeof(acl));
    acl.mChannelLayoutTag = kAudioChannelLayoutTag_Mono;

    NSDictionary *audioOutputSettings = [ NSDictionary dictionaryWithObjectsAndKeys:
            [ NSNumber numberWithInt: kAudioFormatAppleLossless ], AVFormatIDKey,
            [ NSNumber numberWithInt: 16 ], AVEncoderBitDepthHintKey,
            [ NSNumber numberWithFloat: 44100.0 ], AVSampleRateKey,
            [ NSNumber numberWithInt: 1 ], AVNumberOfChannelsKey,
            [ NSData dataWithBytes: &acl length: sizeof( acl ) ], AVChannelLayoutKey,
                    nil ];

    self.assetWriterAudioInput =  [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeAudio
                                                                         outputSettings:audioOutputSettings];

    self.assetWriterAudioInput.expectsMediaDataInRealTime = YES;

    NSParameterAssert(self.assetWriterAudioInput);
    NSParameterAssert([self.assetWriter canAddInput:self.assetWriterAudioInput]);
    [self.assetWriter addInput:self.assetWriterAudioInput];
}

Also need to change minor bit on startRecording warmup code:

- (void)startAssetWriter {
    BOOL aduioReaderStartSuccess = [self.assetAudioReader startReading];
    if(!aduioReaderStartSuccess){
        NSLog(@"asset audio reader start reading failed: %@", self.assetAudioReader.error);
        return;
    }
    BOOL success = [self.assetWriter startWriting];
    if (!success){
        NSLog(@"asset write start writing failed: %@", self.assetWriter.error);
        return;
    }
    [self.assetWriter startSessionAtSourceTime:kCMTimeZero];
    NSLog(@"asset write is good to write...");
}

- (void)startRecording {
    dispatch_async([ContextManager shared].mainSerializationQueue, ^{
        [self setupAssetWriterVideo];
        // Set up the notification that the dispatch group will send when the audio and video work have both finished.
        dispatch_group_notify([ContextManager shared].readingAllReadyDispatchGroup, [ContextManager shared].mainSerializationQueue, ^{
            NSLog(@"all set, readers and writer both are ready");
            [self setupAssetAudioReaderAndWriter];
            [self startAssetWriter];
            [self kickoffRecording];
        });
    });
}

#Audio Writing

Audio writing is quite straightforward:

- (void)kickOffAudioWriting {
    NSAssert(self.recordingDispatchGroup, @"Recording dispatch group should be inited to sync audio/video writing");
    // If there is audio to reencode, enter the dispatch group before beginning the work.
    dispatch_group_enter(self.recordingDispatchGroup);
    // Specify the block to execute when the asset writer is ready for audio media data, and specify the queue to call it on.
    [self.assetWriterAudioInput requestMediaDataWhenReadyOnQueue:[ContextManager shared].rwAudioSerializationQueue usingBlock:^{
        // Because the block is called asynchronously, check to see whether its task is complete.
        if (self.audioFinished)
            return;

        BOOL completedOrFailed = NO;
        // If the task isn't complete yet, make sure that the input is actually ready for more media data.
        while ([self.assetWriterAudioInput isReadyForMoreMediaData] && !completedOrFailed) {
            // Get the next audio sample buffer, and append it to the output file.
            CMSampleBufferRef sampleBuffer = [self.assetAudioReaderTrackOutput copyNextSampleBuffer];
            if (sampleBuffer != NULL) {
                BOOL success = [self.assetWriterAudioInput appendSampleBuffer:sampleBuffer];
                if (success) {
                    NSLog(@"append audio buffer success");
                } else {
                    NSLog(@"append audio buffer failed");
                }
                CFRelease(sampleBuffer);
                sampleBuffer = NULL;
                completedOrFailed = !success;
            }
            else {
                completedOrFailed = YES;
            }


        }//end of loop

        if (completedOrFailed) {
            NSLog(@"audio wrint done");
            // Mark the input as finished, but only if we haven't already done so, and then leave the dispatch group (since the audio work has finished).
            BOOL oldFinished = self.audioFinished;
            self.audioFinished = YES;
            if (!oldFinished) {
                [self.assetWriterAudioInput markAsFinished];
                dispatch_group_leave(self.recordingDispatchGroup);
            };
        }
    }];
}

Okay, we’re good with audio writing, next we need coordinate audio with video.

Audio&&Video Writing Sync

As you may see above, we have used self.recordingDispatchGroup for audio writing, let’s take a look on how it works with audio/video together.

- (void)kickoffRecording {

    // If the asset reader and writer both started successfully, create the dispatch group where the reencoding will take place and start a sample-writing session.
    self.recordingDispatchGroup = dispatch_group_create();
    self.audioFinished = NO;
    self.videoFinished = NO;


    [self kickOffAudioWriting];

    [self kickOffVideoWriting];


    // Set up the notification that the dispatch group will send when the audio and video work have both finished.
    dispatch_group_notify(self.recordingDispatchGroup, [ContextManager shared].mainSerializationQueue, ^{
        self.videoFinished = NO;
        self.audioFinished = NO;
        [self.assetWriter finishWritingWithCompletionHandler:^{
            dispatch_async(dispatch_get_main_queue(), ^{
                if(self.onWritingFinishedBlock){
                    self.onWritingFinishedBlock();
                }
            });
        }];
    });
}

Conclusion

That’s it. Now you’re able to merge audio and video frames from multiple movies.

What’s next is to use OpenGL ES Analyzer to improve the performance.

Video Merging with AVFoundation and OpenGL ES on iOS: Alpha Channel

2014-12-28T14:05:17+00:00

In one of previous post Pixel Perfect Chroma Key Blend Using GPUImage, I have used GPUImage to achieve pixel perfect video blending using alpha channel. But what I did mention is that its performance is not quite stable, and I got run into lots of “problem with pixel appending” logs many times, the final video just doesn’t look right. So I decide to expand my previous code a little to add support for three videoes merging just like GPUImage. And from my experiment,final video looks very perfect comparing to GPUImage. You could see the different on final output vidoes here.

Here is one screenshot of one frame from final video using GPUImage on iPad2:

You could clearly tell that the frame is not perfectly synced.

Here is the one from my implementation:

Note: the complete code is on github: ThreeVidoes-AlphaChannelBlend/. In sample code, I have also implemented how to do chroma key using GPUImage, so that you could compare side by side.

Talk is cheap, show me the damn code.

Shaders

Now we need to change vertext shader to has three input texture coordinates:

attribute vec4 Position;

attribute vec2 srcTexCoordIn1;
attribute vec2 srcTexCoordIn2;
attribute vec2 srcTexCoordIn3;
varying vec2 srcTexCoordOut1;
varying vec2 srcTexCoordOut2;
varying vec2 srcTexCoordOut3;

void main(void) {
    gl_Position = Position;
    srcTexCoordOut1 = srcTexCoordIn1;
    srcTexCoordOut2 = srcTexCoordIn2;
    srcTexCoordOut3 = srcTexCoordIn3;
}    

The fragment shader is quite simple, as we pull the color component from alpha channel video and use it as blend value:

precision highp float;

varying lowp vec2 srcTexCoordOut1;  //alpha
uniform sampler2D srcTexture1; 

varying lowp vec2 srcTexCoordOut2;  //fx
uniform sampler2D srcTexture2;  

varying lowp vec2 srcTexCoordOut3;  //raw/src
uniform sampler2D srcTexture3;  

uniform float thresholdSensitivity;
uniform float smoothing;
uniform vec3 colorToReplace;

void main(void) {
    vec4 textureColorAlpha = texture2D(srcTexture1, srcTexCoordOut1);//alpha
    vec4 textureColorFX = texture2D(srcTexture2, srcTexCoordOut2); //fx
    vec4 textureColorSrc = texture2D(srcTexture3, srcTexCoordOut3); //src

    gl_FragColor = mix(textureColorFX, textureColorSrc, 1.0 -textureColorAlpha.r);

}

Setup

fxURL = [[NSBundle mainBundle] URLForResource:@"fireworks_sd" withExtension:@"mp4"];
alphaURL = [[NSBundle mainBundle] URLForResource:@"fireworks_alpha_sd" withExtension:@"mp4"];
rawURL = [[NSBundle mainBundle] URLForResource:@"video" withExtension:@"mov"];


videoWriter = [[VideoWriter alloc] init];
videoReaderAlpha = [[VideoReader alloc] initWithURL:alphaURL];
videoReaderFX = [[VideoReader alloc] initWithURL:fxURL];
videoReaderRaw = [[VideoReader alloc] initWithURL:rawURL];

videoWriter.readerAlpha = videoReaderAlpha;
videoWriter.readerRaw = videoReaderRaw;
videoWriter.readerFX = videoReaderFX;

[videoReaderAlpha startProcessing];
[videoReaderRaw startProcessing];
[videoReaderFX startProcessing];
[videoWriter startRecording];

Okay, we’re good with sample code, so what change we need on video reader and writer ?

Yeah, it turns out it is like what we did in two video merging: video reader doesn’t need any change and we only need to change VideoWriter to add support for three readers.

VideoWriter

First, we need to change **- (void)compileShaders ** method to get handler for third texture(because of one more video).

    _positionSlot = glGetAttribLocation(_program, "Position");
    _srcTexCoord1Slot = glGetAttribLocation(_program, "srcTexCoordIn1");
    _srcTexCoord2Slot = glGetAttribLocation(_program, "srcTexCoordIn2");
    _srcTexCoord3Slot = glGetAttribLocation(_program, "srcTexCoordIn3");

    glEnableVertexAttribArray(_positionSlot);
    glEnableVertexAttribArray(_srcTexCoord1Slot);
    glEnableVertexAttribArray(_srcTexCoord2Slot);
    glEnableVertexAttribArray(_srcTexCoord3Slot);

    _thresholdUniform = glGetUniformLocation(_program, "thresholdSensitivity");
    _smoothingUniform = glGetUniformLocation(_program, "smoothing");
    _colorToReplaceUniform = glGetUniformLocation(_program, "colorToReplace");

    _srcTexture1Uniform = glGetUniformLocation(_program, "srcTexture1");
    _srcTexture2Uniform = glGetUniformLocation(_program, "srcTexture2");
    _srcTexture3Uniform = glGetUniformLocation(_program, "srcTexture3");

For kickoffRecording, it is quite simple now:

- (void)kickoffRecording {
    [self.assetWriterVideoInput requestMediaDataWhenReadyOnQueue:UTIL.rwVideoSerializationQueue usingBlock:^{
        BOOL completedOrFailed = NO;
        // If the task isn't complete yet, make sure that the input is actually ready for more media data.
        while ([self.assetWriterVideoInput isReadyForMoreMediaData] && !completedOrFailed)
        {
            // Get the next video sample buffer, and append it to the output file.
            dispatch_group_t downloadGroup = dispatch_group_create(); // 2

            __block FrameRenderOutput *alphaFrameOutput;
            dispatch_group_async(downloadGroup, dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
                alphaFrameOutput = [self.readerAlpha renderNextFrame];
            });

            __block FrameRenderOutput *fxFrameOutput;
            dispatch_group_async(downloadGroup, dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
                fxFrameOutput = [self.readerFX renderNextFrame];
            });

            __block FrameRenderOutput *rawFrameOutput;
            dispatch_group_async(downloadGroup, dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
                rawFrameOutput = [self.readerRaw renderNextFrame];
            });



            NSLog(@"wait is starting");
            dispatch_group_wait(downloadGroup, DISPATCH_TIME_FOREVER); // 5
            NSLog(@"wait is done");
            if(!fxFrameOutput.sampleBuffer || !rawFrameOutput.sampleBuffer || !alphaFrameOutput.sampleBuffer){
                //reading done
                completedOrFailed = YES;
            } else {
                NSLog(@"------------ready-------recevied both:%d", fxFrameOutput.outputTexture);

                CVPixelBufferLockBaseAddress(_pixelBuffer, 0);

                [[ContextManager shared] useCurrentContext];

                if(_frameBuffer == 0){
                    [self createFrameBufferObject];

                }
                glBindFramebuffer(GL_FRAMEBUFFER, _frameBuffer);

                glClearColor(0.0, 0.0, 0.0, 1.0);
                glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

                glViewport(0, 0, (int)self.videoSize.width, (int)self.videoSize.height);
                //use shader program
                NSAssert(_program, @"Program should be created");
                glUseProgram(_program);


                // This needs to be flipped to write out to video correctly
                static const GLfloat squareVertices[] = {
                        -1.0f, -1.0f,
                        1.0f, -1.0f,
                        -1.0f,  1.0f,
                        1.0f,  1.0f,
                };

                static const GLfloat textureCoordinates[] = {
                        0.0f, 0.0f,
                        1.0f, 0.0f,
                        0.0f, 1.0f,
                        1.0f, 1.0f,
                };
                glVertexAttribPointer(_positionSlot, 2, GL_FLOAT, 0, 0, squareVertices);
                glVertexAttribPointer(_srcTexCoord1Slot, 2, GL_FLOAT, 0, 0, textureCoordinates);
                glVertexAttribPointer(_srcTexCoord2Slot, 2, GL_FLOAT, 0, 0, textureCoordinates);
                glVertexAttribPointer(_srcTexCoord3Slot, 2, GL_FLOAT, 0, 0, textureCoordinates);

                //bind uniforms
                glUniform1f(_thresholdUniform, 0.4f);
                glUniform1f(_smoothingUniform, 0.1f);

                Vector3 colorToReplaceVec3 = {0.0f, 1.0f, 0.0f};
                glUniform3fv(_colorToReplaceUniform, 1, (GLfloat *)&colorToReplaceVec3);

                glActiveTexture(GL_TEXTURE2);
                glBindTexture(GL_TEXTURE_2D, alphaFrameOutput.outputTexture);
                glUniform1i(_srcTexture1Uniform, 2);

                glActiveTexture(GL_TEXTURE3);
                glBindTexture(GL_TEXTURE_2D, fxFrameOutput.outputTexture);
                glUniform1i(_srcTexture2Uniform, 3);

                glActiveTexture(GL_TEXTURE4);
                glBindTexture(GL_TEXTURE_2D, rawFrameOutput.outputTexture);
                glUniform1i(_srcTexture3Uniform, 4);

                glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
                glFinish();

                CMTime frameTime = fxFrameOutput.frameTime;

                if(CMTimeCompare(frameTime, lastFrameTime) == NSOrderedSame){
                    NSLog(@"***********************FATAL ERROR, frame times are same");
                }

                //CVPixelBufferLockBaseAddress(_pixelBuffer, 0);

                BOOL writeSucceeded = [self.assetWriterPixelBufferInput appendPixelBuffer:_pixelBuffer withPresentationTime:frameTime];

                CVPixelBufferUnlockBaseAddress(_pixelBuffer, 0);

                if(writeSucceeded){
                    NSLog(@"==================dWrote a video frame: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, frameTime)));
                    lastFrameTime = frameTime;
                }else{
                    //  NSLog(@"pixel buffer pool : %@", assetWriterPixelBufferInput.pixelBufferPool);
                    NSLog(@"Problem appending pixel buffer at time: %@ with error: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, frameTime)), self.assetWriter.error);
                    lastFrameTime = frameTime;

                }
                [self.readerAlpha cleanupResource:alphaFrameOutput];
                [self.readerFX cleanupResource:fxFrameOutput];
                [self.readerRaw cleanupResource:rawFrameOutput];
            }

        }
        if (completedOrFailed)
        {
            NSLog(@"mark as finish");
            // Mark the input as finished, but only if we haven't already done so, and then leave the dispatch group (since the video work has finished).
            [self.assetWriterVideoInput markAsFinished];
            [self.assetWriter finishWritingWithCompletionHandler:^{
                dispatch_async(dispatch_get_main_queue(), ^{
                    [SVProgressHUD showWithStatus:@"Write done"];
                    if(self.onWritingFinishedBlock){
                        self.onWritingFinishedBlock(self.outputURL);
                    }
                });
            }];
        }
    }];
}

Here we used dispatch_group_async to dispatch reading/uploading frame currently to get better performance as what we did in two video merging. After three frames rendered, then we pass it to our shader program, you have to pay attention to this code part:

            glActiveTexture(GL_TEXTURE2);
            glBindTexture(GL_TEXTURE_2D, alphaFrameOutput.outputTexture);
            glUniform1i(_srcTexture1Uniform, 2);

            glActiveTexture(GL_TEXTURE3);
            glBindTexture(GL_TEXTURE_2D, fxFrameOutput.outputTexture);
            glUniform1i(_srcTexture2Uniform, 3);

            glActiveTexture(GL_TEXTURE4);
            glBindTexture(GL_TEXTURE_2D, rawFrameOutput.outputTexture);
            glUniform1i(_srcTexture3Uniform, 4);

Again, the order is very important.

That’s it, that’s all you need to change to make it support three video merging to pixel perfect chroma key blending :)

What’s Next

Audio Tracks Merge: We already have video merged, but not for audio part. So in next article, we will add function to merge audio tracks from all vidoes: .
Performance Optimization: All above test is based on 640640 resolution, what if we try 1280720 resolution videos? We will try to use OpenGL ES Analyzer in Instrument to help improve the performance.

Video Merging with AVFoundation and OpenGL ES on iOS: Chroma Key

2014-12-28T13:27:57+00:00

The goal of this article is to take a input video(e.g. green background video), and a raw video, then blend fx video over raw video by masking out green background color, which is also called chroma key.

To achieve this, we need to modify the code that we had for one video transform little bit.

Note: the complete code is on github: TwoVideoes-ChromaKey. In sample code, I have also implemented how to do chroma key using GPUImage, so that you could compare side by side.

#Shaders

Now we need to change vertext shader to has two input texture coordinates:

attribute vec4 Position;

attribute vec2 srcTexCoordIn1;
attribute vec2 srcTexCoordIn2;
varying vec2 srcTexCoordOut1;
varying vec2 srcTexCoordOut2;

void main(void) {
    gl_Position = Position;
    srcTexCoordOut1 = srcTexCoordIn1;
    srcTexCoordOut2 = srcTexCoordIn2;
}

The fragment shader we take from GPUImage’s GPUImageChromaKeyBlendFilter:

precision highp float;

varying lowp vec2 srcTexCoordOut1; // Raw
uniform sampler2D srcTexture1; // Raw

varying lowp vec2 srcTexCoordOut2; // FX
uniform sampler2D srcTexture2; // FX

uniform float thresholdSensitivity;
uniform float smoothing;
uniform vec3 colorToReplace;

void main(void) { 
    vec4 textureColor = texture2D(srcTexture1, srcTexCoordOut1); //raw     
    vec4 textureColor1 = texture2D(srcTexture2, srcTexCoordOut2); //fx

    float maskY = 0.2989 * colorToReplace.r + 0.5866 * colorToReplace.g + 0.1145 * colorToReplace.b;
    float maskCr = 0.7132 * (colorToReplace.r - maskY);
    float maskCb = 0.5647 * (colorToReplace.b - maskY);

    float Y = 0.2989 * textureColor.r + 0.5866 * textureColor.g + 0.1145 * textureColor.b;
    float Cr = 0.7132 * (textureColor.r - Y);
    float Cb = 0.5647 * (textureColor.b - Y);

    //     float blendValue = 1.0 - smoothstep(thresholdSensitivity - smoothing, thresholdSensitivity , abs(Cr - maskCr) + abs(Cb - maskCb));
    float blendValue = 1.0 - smoothstep(thresholdSensitivity, thresholdSensitivity + smoothing, distance(vec2(Cr, Cb), vec2(maskCr, maskCb)));
    //gl_FragColor = textureColor; //set it red for testing
    gl_FragColor = mix(textureColor, textureColor1, blendValue);

}

So for anyone who has used GPUImage to do chromakey blend, this should look very familiar.

Setup

rawURL = [[NSBundle mainBundle] URLForResource:@"video" withExtension:@"mov"];
fxURL = [[NSBundle mainBundle] URLForResource:@"FXSample" withExtension:@"mov"];

videoWriter = [[VideoWriter alloc] init];
videoReaderRaw = [[VideoReader alloc] initWithURL:rawURL];
videoReaderFX = [[VideoReader alloc] initWithURL:fxURL];

videoWriter.readerRaw = videoReaderRaw;
videoWriter.readerFX = videoReaderFX;

[videoReaderRaw startProcessing];
[videoReaderFX startProcessing];
[videoWriter startRecording];

Okay, we’re good with sample code, so what change we need on video reader and writer ?

Well, it turns out video reader doesn’t need any change. We only need to change VideoWriter to add support for two readers.

VideoWriter

First, we need to change **- (void)compileShaders ** method to get handler for second texture(because of one more video).

    _positionSlot = glGetAttribLocation(_program, "Position");
    _srcTexCoord1Slot = glGetAttribLocation(_program, "srcTexCoordIn1");
    _srcTexCoord2Slot = glGetAttribLocation(_program, "srcTexCoordIn2");

    glEnableVertexAttribArray(_positionSlot);
    glEnableVertexAttribArray(_srcTexCoord1Slot);
    glEnableVertexAttribArray(_srcTexCoord2Slot);

    _thresholdUniform = glGetUniformLocation(_program, "thresholdSensitivity");
    _smoothingUniform = glGetUniformLocation(_program, "smoothing");
    _colorToReplaceUniform = glGetUniformLocation(_program, "colorToReplace");

    _srcTexture1Uniform = glGetUniformLocation(_program, "srcTexture1");
    _srcTexture2Uniform = glGetUniformLocation(_program, "srcTexture2");

Cool, so the setup is okay, let’s move to most important part: kickoffRecording.

- (void)kickoffRecording {
    [self.assetWriterVideoInput requestMediaDataWhenReadyOnQueue:UTIL.rwVideoSerializationQueue usingBlock:^{
        BOOL completedOrFailed = NO;
        // If the task isn't complete yet, make sure that the input is actually ready for more media data.
        while ([self.assetWriterVideoInput isReadyForMoreMediaData] && !completedOrFailed)
        {
            // Get the next video sample buffer, and append it to the output file.
            dispatch_group_t downloadGroup = dispatch_group_create(); // 2

            __block FrameRenderOutput *fxFrameOutput;
            dispatch_group_async(downloadGroup, dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
                fxFrameOutput = [self.readerFX renderNextFrame];
            });

            __block FrameRenderOutput *rawFrameOutput;
            dispatch_group_async(downloadGroup, dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
                rawFrameOutput = [self.readerRaw renderNextFrame];
            });
            NSLog(@"wait is starting");
            dispatch_group_wait(downloadGroup, DISPATCH_TIME_FOREVER); // 5
            NSLog(@"wait is done");
            if(!fxFrameOutput.sampleBuffer || !rawFrameOutput.sampleBuffer){
                //reading done
                completedOrFailed = YES;
            } else {
                NSLog(@"------------ready-------recevied both:%d", fxFrameOutput.outputTexture);

                CVPixelBufferLockBaseAddress(_pixelBuffer, 0);

                [[ContextManager shared] useCurrentContext];

                if(_frameBuffer == 0){
                    [self createFrameBufferObject];

                }
                glBindFramebuffer(GL_FRAMEBUFFER, _frameBuffer);

                glClearColor(0.0, 0.0, 0.0, 1.0);
                glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

                glViewport(0, 0, (int)self.videoSize.width, (int)self.videoSize.height);
                //use shader program
                NSAssert(_program, @"Program should be created");
                glUseProgram(_program);


                // This needs to be flipped to write out to video correctly
                static const GLfloat squareVertices[] = {
                        -1.0f, -1.0f,
                        1.0f, -1.0f,
                        -1.0f,  1.0f,
                        1.0f,  1.0f,
                };

                static const GLfloat textureCoordinates[] = {
                        0.0f, 0.0f,
                        1.0f, 0.0f,
                        0.0f, 1.0f,
                        1.0f, 1.0f,
                };
                glVertexAttribPointer(_positionSlot, 2, GL_FLOAT, 0, 0, squareVertices);
                glVertexAttribPointer(_srcTexCoord1Slot, 2, GL_FLOAT, 0, 0, textureCoordinates);
                glVertexAttribPointer(_srcTexCoord2Slot, 2, GL_FLOAT, 0, 0, textureCoordinates);

                //bind uniforms
                glUniform1f(_thresholdUniform, 0.4f);
                glUniform1f(_smoothingUniform, 0.1f);

                Vector3 colorToReplaceVec3 = {0.0f, 1.0f, 0.0f};
                glUniform3fv(_colorToReplaceUniform, 1, (GLfloat *)&colorToReplaceVec3);

                glActiveTexture(GL_TEXTURE2);
                glBindTexture(GL_TEXTURE_2D, rawFrameOutput.outputTexture);
                glUniform1i(_srcTexture1Uniform, 2);

                glActiveTexture(GL_TEXTURE3);
                glBindTexture(GL_TEXTURE_2D, fxFrameOutput.outputTexture);
                glUniform1i(_srcTexture2Uniform, 3);

                glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
                glFinish();

                CMTime frameTime = fxFrameOutput.frameTime;

                if(CMTimeCompare(frameTime, lastFrameTime) == NSOrderedSame){
                    NSLog(@"***********************FATAL ERROR, frame times are same");
                }

                BOOL writeSucceeded = [self.assetWriterPixelBufferInput appendPixelBuffer:_pixelBuffer withPresentationTime:frameTime];
                CVPixelBufferUnlockBaseAddress(_pixelBuffer, 0);

                if(writeSucceeded){
                    NSLog(@"==================dWrote a video frame: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, frameTime)));
                    lastFrameTime = frameTime;
                }else{
                    //  NSLog(@"pixel buffer pool : %@", assetWriterPixelBufferInput.pixelBufferPool);
                    NSLog(@"Problem appending pixel buffer at time: %@ with error: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, frameTime)), self.assetWriter.error);

                    lastFrameTime = frameTime;
                }

                [self.readerFX cleanupResource:fxFrameOutput];
                [self.readerRaw cleanupResource:rawFrameOutput];
            }

        }
        if (completedOrFailed)
        {
            NSLog(@"mark as finish");
            // Mark the input as finished, but only if we haven't already done so, and then leave the dispatch group (since the video work has finished).
            [self.assetWriterVideoInput markAsFinished];
            [self.assetWriter finishWritingWithCompletionHandler:^{
                dispatch_async(dispatch_get_main_queue(), ^{
                    [SVProgressHUD showWithStatus:@"Write done"];
                    if(self.onWritingFinishedBlock){
                        self.onWritingFinishedBlock(self.outputURL);
                    }
                });
            }];
        }
    }];
}

Here we used dispatch_group_async to dispatch reading/uploading frame currently to get better performance. After both frames rendered, then we pass it to our shader program, you have to pay attention to this code part:

            glActiveTexture(GL_TEXTURE2);
            glBindTexture(GL_TEXTURE_2D, rawFrameOutput.outputTexture);
            glUniform1i(_srcTexture1Uniform, 2);

            glActiveTexture(GL_TEXTURE3);
            glBindTexture(GL_TEXTURE_2D, fxFrameOutput.outputTexture);
            glUniform1i(_srcTexture2Uniform, 3);

The order is very important, as it is mapped to the order we defined in our shader, which could lead to completely different result if it is wrong.

That’s it, that’s all you need to change to make it support two video merging with chroma key.

Video Merging with AVFoundation and OpenGL ES on iOS: One Video

2014-12-26T15:36:11+00:00

The goal of this article is to take a input video(e.g. green background video), do some magic in glsl(e.g.remove green color component), then write the output to a video on disk.

To illustrate that:

Before

After

Note: this will invovles lots of coding, I will break into reasonable section with comments to make it clear.Before we jump into the code, I highly recommend you to watch WWDC 2011 video 419 ‘Capturing from the Camera using AVFoundation on iOS 5’, which covers how to combine AVFoundation with OpenGL ES so that it would be much easier for you to understand what following is about.

Also you could find/download/run the sample project/code in github: AVFoundationOpenGLESVideoProcessing: OneVideo

Code Walkthrough

We will have a video reader and video writer class to carray each’s responsibility.

For video reader class, its responsiblities are:

pass in video url
setup CVOpenGLESTextureCache
init asset and load tracks asyncronously when trigger startProcess is called
after tracks are loaded, setup AssetReader
kick off asset reader, now the reader is ready to pull out of frames, which is gonna be inited from video writer

For video writer class, its responsiblities are:

on init, setupOpenGLESTextureCache
compile shaders(vertext/fragment,create program, link, get attributes/uniform handlers)
when being triggered with startRecording, setup AssetWriter, pay close attention to AVAssetWriterInputPixelBufferAdaptor
start asset writer writing, now writer is ready
kick off recording

In step5(kick off recording), we will pull each frame from reader, then upload it to GPU with returned texture pointer, pass that pointer to GPU with name we defined in GLSL, render it, get back the pixel buffers mapped from final output of GPU, write that pixel buffer to our final destination video.

Setup

fxURL = [[NSBundle mainBundle] URLForResource:@"FXSample" withExtension:@"mov"];
videoWriter = [[VideoWriter alloc] init];
videoReaderFX = [[VideoReader alloc] initWithURL:fxURL];

[videoReaderFX startProcessing];
[videoWriter startRecording];

Video Reader Setup

setup code:

- (id)initWithURL:(NSURL *)pUrl{
    self = [super init]; 
    if (self) {
        self.url = pUrl;       
        [self setupOpenGLESTextureCache];
    }
    return self;
}


- (void)setupOpenGLESTextureCache {
    // Create a new CVOpenGLESTexture cache
    //-- Create CVOpenGLESTextureCacheRef for optimal CVImageBufferRef to GLES texture conversion.
#if COREVIDEO_USE_EAGLCONTEXT_CLASS_IN_API
    CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, [ContextManager shared].currentContext, NULL, &coreVideoTextureCache);
#else
        CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, (__bridge void *)([ContextManager shared].currentContext), NULL, &coreVideoTextureCache);
    #endif
    if (err) {
        NSLog(@"Error at CVOpenGLESTextureCacheCreate %d", err);
        exit(1);
    }
}

Load video and prepare:

- (void)setupAssetReader {
   
    NSError* error;
    self.assetReader = [AVAssetReader assetReaderWithAsset:self.asset error:&error];
    NSParameterAssert(self.assetReader);


    NSDictionary *outputSettings = @{
            (id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA)
    };

    NSArray *videoTracks = [self.asset tracksWithMediaType:AVMediaTypeVideo];
    AVAssetTrack *videoTrack = [videoTracks objectAtIndex:0];

    self.assetReaderOutput = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:videoTrack
                                                                     outputSettings:outputSettings];

    NSParameterAssert(self.assetReaderOutput);
    NSParameterAssert([self.assetReader canAddOutput:self.assetReaderOutput]);
    [self.assetReader addOutput:self.assetReaderOutput];
    
}

- (void)startAssetReader {
    __weak typeof(self) weakSelf = self;
    if (![self.assetReader startReading])
    {
        NSLog(@"Error reading from file at URL: %@", self.url);
        return;
    }

    NSLog(@"asset %@ is good to read...", self.url);
}


- (void)startProcessing{
    dispatch_group_enter(UTIL.dispatchGroup);
    self.asset = [AVAsset assetWithURL:_url];
    CMTime duration = self.asset.duration;
    CGFloat durationInSeconds = (CGFloat) CMTimeGetSeconds(duration);
    __weak typeof(self) weakSelf = self;
    [self.asset loadValuesAsynchronouslyForKeys:@[@"tracks"] completionHandler:^{
        // Once the tracks have finished loading, dispatch the work to the main serialization queue.
        dispatch_async(UTIL.mainSerializationQueue, ^{
            BOOL success = YES;
            NSError *localError = nil;
            // Check for success of loading the assets tracks.
            success = ([self.asset statusOfValueForKey:@"tracks" error:&localError] == AVKeyValueStatusLoaded);
            if (success)
            {
            }
            if (success){
                [self setupAssetReader];
            }
            if (success){
                [self startAssetReader];
            }

            dispatch_group_leave(UTIL.dispatchGroup);
        });
    }];
}

Video Writer Setup

Note: video writer setup and warm up is more complicted than reader slightly.

First part is setup code:

- (void)setup {

    self.videoSize = CGSizeMake(640, 640);
    [self setupOpenGLESTextureCache];
    [self compileShaders];
}

- (void)setupOpenGLESTextureCache {
    // Create a new CVOpenGLESTexture cache
    //-- Create CVOpenGLESTextureCacheRef for optimal CVImageBufferRef to GLES texture conversion.
#if COREVIDEO_USE_EAGLCONTEXT_CLASS_IN_API
    CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, [ContextManager shared].currentContext, NULL, &_textureCache);
#else
        CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, (__bridge void *)([ContextManager shared].currentContext), NULL, &_textureCache);
    #endif
    if (err) {
        NSLog(@"Error at CVOpenGLESTextureCacheCreate %d", err);
        exit(1);
    }
}


- (void)compileShaders {

    // 1
    GLuint vertexShader = [self compileShader:VERTEX_SHADER_FILENAME withType:GL_VERTEX_SHADER];
    GLuint fragmentShader = [self compileShader:FRAGMENT_SHADER_FILENAME withType:GL_FRAGMENT_SHADER];

    // 2
    _program = glCreateProgram();
    glAttachShader(_program, vertexShader);
    glAttachShader(_program, fragmentShader);
    glLinkProgram(_program);

    // 3
    GLint linkSuccess;
    glGetProgramiv(_program, GL_LINK_STATUS, &linkSuccess);
    if (linkSuccess == GL_FALSE) {
        GLchar messages[256];
        glGetProgramInfoLog(_program, sizeof(messages), 0, &messages[0]);
        NSString *messageString = [NSString stringWithUTF8String:messages];
        NSLog(@"%@", messageString);
        exit(1);
    }


    // 4
    _positionSlot = glGetAttribLocation(_program, "Position");
    _srcTexCoord1Slot = glGetAttribLocation(_program, "srcTexCoordIn1");

    glEnableVertexAttribArray(_positionSlot);
    glEnableVertexAttribArray(_srcTexCoord1Slot);

    _thresholdUniform = glGetUniformLocation(_program, "thresholdSensitivity");
    _smoothingUniform = glGetUniformLocation(_program, "smoothing");
    _colorToReplaceUniform = glGetUniformLocation(_program, "colorToReplace");

    _srcTexture1Uniform = glGetUniformLocation(_program, "srcTexture1");
}

- (GLuint)compileShader:(NSString*)shaderName withType:(GLenum)shaderType {
    //... 
}

First part is setup code:

- (void)setupAssetWriter {
    // Do any additional setup after loading the view.
    NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
    NSString *documentsDirectory = [paths objectAtIndex:0];
//    NSString *myPathDocs =  [documentsDirectory stringByAppendingPathComponent:@"merge-output.mov"];
    NSString *myPathDocs =  [documentsDirectory stringByAppendingPathComponent:
            [NSString stringWithFormat:@"merge-output-%d.mov",arc4random() % 1000]];
    self.outputURL = [NSURL fileURLWithPath:myPathDocs];

    // If the tracks loaded successfully, make sure that no file exists at the output path for the asset writer.
    NSFileManager *fm = [NSFileManager defaultManager];
    NSString *localOutputPath = [self.outputURL path];
    if ([fm fileExistsAtPath:localOutputPath]){
        [fm removeItemAtPath:localOutputPath error:nil];
    }

    NSLog(@"setupAssetWriter outputURL: %@", self.outputURL );

    self.movieWritingQueue = dispatch_queue_create("tuo.test.movieWritingQueue", NULL);


    NSError *error;
    self.assetWriter = [AVAssetWriter assetWriterWithURL:self.outputURL
                                                fileType:AVFileTypeQuickTimeMovie error:&error];

    NSParameterAssert(self.assetWriter);



    NSDictionary *videoSettings = @{
            AVVideoCodecKey : AVVideoCodecH264,
            AVVideoWidthKey : @(self.videoSize.width),
            AVVideoHeightKey : @(self.videoSize.height)
    };
    self.assetWriterVideoInput =  [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo
                                                                     outputSettings:videoSettings];
    // You need to use BGRA for the video in order to get realtime encoding. I use a color-swizzling shader to line up glReadPixels' normal RGBA output with the movie input's BGRA.
    NSDictionary *sourcePixelBufferAttributesDictionary = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey,
                                                                                                      [NSNumber numberWithInt:self.videoSize.width], kCVPixelBufferWidthKey,
                                                                                                      [NSNumber numberWithInt:self.videoSize.height], kCVPixelBufferHeightKey,
                    nil];

    self.assetWriterPixelBufferInput = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:self.assetWriterVideoInput
                                                                                                        sourcePixelBufferAttributes:sourcePixelBufferAttributesDictionary];

    NSParameterAssert(self.assetWriterVideoInput);
    NSParameterAssert([self.assetWriter canAddInput:self.assetWriterVideoInput]);
    [self.assetWriter addInput:self.assetWriterVideoInput]; 
}

- (void)startAssetWriter {
    BOOL success = [self.assetWriter startWriting];
    if (!success){
        NSLog(@"asset write start writing failed: %@", self.assetWriter.error);
        return;
    }
    [self.assetWriter startSessionAtSourceTime:kCMTimeZero];
    NSLog(@"asset write is good to write...");
}

- (void)startRecording {
    dispatch_async(UTIL.mainSerializationQueue, ^{
        [self setupAssetWriter];
        [self startAssetWriter];

        // Set up the notification that the dispatch group will send when the audio and video work have both finished.
        dispatch_group_notify(UTIL.dispatchGroup, UTIL.mainSerializationQueue, ^{
            NSLog(@"all set, readers and writer both are ready");

            [self kickoffRecording];
        });
    });
}

Video Writer KickOff

This is key part of whole thing:

- (void)kickoffRecording {
    [self.assetWriterVideoInput requestMediaDataWhenReadyOnQueue:UTIL.rwVideoSerializationQueue usingBlock:^{
        BOOL completedOrFailed = NO;
        // If the task isn't complete yet, make sure that the input is actually ready for more media data.
        while ([self.assetWriterVideoInput isReadyForMoreMediaData] && !completedOrFailed)
        {

            FrameRenderOutput *fxFrameOutput = [self.readerFX renderNextFrame];

            if(!fxFrameOutput.sampleBuffer){
                //reading done
                completedOrFailed = YES;
            } else {

                CVPixelBufferLockBaseAddress(_pixelBuffer, 0);

                [[ContextManager shared] useCurrentContext];

                if(_frameBuffer == 0){
                    [self createFrameBufferObject];

                }
                glBindFramebuffer(GL_FRAMEBUFFER, _frameBuffer);

                glClearColor(0.0, 0.0, 0.0, 1.0);
                glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

                glViewport(0, 0, (int)self.videoSize.width, (int)self.videoSize.height);
                //use shader program
                NSAssert(_program, @"Program should be created");
                glUseProgram(_program);


                // This needs to be flipped to write out to video correctly
                static const GLfloat squareVertices[] = {
                        -1.0f, -1.0f,
                        1.0f, -1.0f,
                        -1.0f,  1.0f,
                        1.0f,  1.0f,
                };

                static const GLfloat textureCoordinates[] = {
                        0.0f, 0.0f,
                        1.0f, 0.0f,
                        0.0f, 1.0f,
                        1.0f, 1.0f,
                };
                glVertexAttribPointer(_positionSlot, 2, GL_FLOAT, 0, 0, squareVertices);
                glVertexAttribPointer(_srcTexCoord1Slot, 2, GL_FLOAT, 0, 0, textureCoordinates);

                //bind uniforms
                glUniform1f(_thresholdUniform, 0.4f);
                glUniform1f(_smoothingUniform, 0.1f);

                Vector3 colorToReplaceVec3 = {0.0f, 1.0f, 0.0f};
                glUniform3fv(_colorToReplaceUniform, 1, (GLfloat *)&colorToReplaceVec3);

                glActiveTexture(GL_TEXTURE2);
                glBindTexture(GL_TEXTURE_2D, fxFrameOutput.outputTexture);
                glUniform1i(_srcTexture1Uniform, 2);


                glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
                glFinish();
                
                CMTime frameTime = fxFrameOutput.frameTime;

                if(CMTimeCompare(frameTime, lastFrameTime) == NSOrderedSame){
                    NSLog(@"***********************FATAL ERROR, frame times are same");
                }

                BOOL writeSucceeded = [self.assetWriterPixelBufferInput appendPixelBuffer:_pixelBuffer withPresentationTime:frameTime];
                CVPixelBufferUnlockBaseAddress(_pixelBuffer, 0);
                if(writeSucceeded){
                    lastFrameTime = frameTime;
                }else{
                    NSLog(@"Problem appending pixel buffer at time: %@ with error: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, frameTime)), self.assetWriter.error);

                }
                [self.readerFX cleanupResource:fxFrameOutput];
            }

        }
        if (completedOrFailed)
        {
            NSLog(@"mark as finish");
            // Mark the input as finished, but only if we haven't already done so, and then leave the dispatch group (since the video work has finished).
            [self.assetWriterVideoInput markAsFinished];
            [self.assetWriter finishWritingWithCompletionHandler:^{
                dispatch_async(dispatch_get_main_queue(), ^{
                    [SVProgressHUD showWithStatus:@"Write done"];
                    if(self.onWritingFinishedBlock){
                        self.onWritingFinishedBlock(self.outputURL);
                    }
                });
            }];
        }
    }];
}

The renderNextFrame in video reader shows how to read frame and upload to GPU:

- (FrameRenderOutput *)renderNextFrame{

    CMSampleBufferRef sampleBuffer = [self.avReaderOutput copyNextSampleBuffer];
    NSLog(@"%@, sampleBuffer: %d, reader status: %d", self.url.lastPathComponent, sampleBuffer != nil, self.assetReader.status);

    if(sampleBuffer == nil){
        NSLog(@"%@ ----- reading done", self.url.lastPathComponent);
        FrameRenderOutput *frameRenderOutput = [[FrameRenderOutput alloc] init];
        frameRenderOutput.sampleBuffer = nil;
        return frameRenderOutput;
    }


    CMTime currentSampleTime = CMSampleBufferGetOutputPresentationTimeStamp(sampleBuffer);
    CVImageBufferRef movieFrame = CMSampleBufferGetImageBuffer(sampleBuffer);


    [[ContextManager shared] useCurrentContext];

    CVPixelBufferLockBaseAddress(movieFrame, 0);

    int bufferHeight = CVPixelBufferGetHeight(movieFrame);
    int bufferWidth = CVPixelBufferGetWidth(movieFrame);

    if(!framebuffer){
        glGenFramebuffers(1, &framebuffer);
        glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
    }
    glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);



    CVReturn err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
            coreVideoTextureCache,
            movieFrame,
            NULL,
            GL_TEXTURE_2D,
            GL_RGBA,
            bufferWidth,
            bufferHeight,
            GL_BGRA,
            GL_UNSIGNED_BYTE,
            0,
            &texture);

    if (!texture || err) {
        NSLog(@"Movie CVOpenGLESTextureCacheCreateTextureFromImage failed (error: %d)", err);
        NSAssert(NO, @"Camera failure");
        return nil;
    }

    outputTexture = CVOpenGLESTextureGetName(texture);
    //        glBindTexture(CVOpenGLESTextureGetTarget(texture), outputTexture);
    glBindTexture(GL_TEXTURE_2D, outputTexture);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, CVOpenGLESTextureGetName(texture), 0);


    FrameRenderOutput *frameRenderOutput = [[FrameRenderOutput alloc] init];
    frameRenderOutput.outputTexture = outputTexture;
    frameRenderOutput.inputTexture = texture;
    frameRenderOutput.frameTime = currentSampleTime;
    frameRenderOutput.sampleBuffer = sampleBuffer;

    NSLog(@"%@, frameRenderOutput: %@", self.url.lastPathComponent, frameRenderOutput);
    return frameRenderOutput;
}

- (void)cleanupResource:(FrameRenderOutput *)output {
    CVOpenGLESTextureCacheFlush(coreVideoTextureCache, 0);
    CFRelease(texture);

    NSLog(@"clean up resources: %@", self.url.lastPathComponent);
    CMSampleBufferInvalidate(output.sampleBuffer);
    CFRelease(output.sampleBuffer);
    output.sampleBuffer = NULL;

    outputTexture = 0;

}

FrameRenderOutput is just simply data object to hold some intermediate information about frame:

@interface FrameRenderOutput : NSObject

@property (nonatomic) GLuint outputTexture;
@property (nonatomic) CMTime frameTime;

@property(nonatomic) CMSampleBufferRef sampleBuffer;

@property(nonatomic) CVOpenGLESTextureRef inputTexture;

- (NSString *)description;


@end

Shader GLSL

Here is the vertext file:

attribute vec4 Position;

attribute vec2 srcTexCoordIn1;
varying vec2 srcTexCoordOut1;

void main(void) {
    gl_Position = Position;
    srcTexCoordOut1 = srcTexCoordIn1;
}

very simple. And fragment shader is :

precision highp float;

varying lowp vec2 srcTexCoordOut1; // New
uniform sampler2D srcTexture1; // New

void main(void) {
    vec4 textureColor = texture2D(srcTexture1, srcTexCoordOut1); //movie fx video

    gl_FragColor = vec4(textureColor.r,0, textureColor.b, textureColor.a);

}

That’s it! Give it a run, you should be able to see final result.

Video Merging with AVFoundation and OpenGL ES on iOS: Introduction

2014-12-26T12:00:46+00:00

Video merging on iOS is kinda tricky even though there is good library like GPUImage there. If you look at the issue page of GPUImage, you would realize lots of the issues relates to GPUImageMovie or GPUImageMovieWriter, yes, it is very good at image processing, but for video, I’m not sure about that.

Problems

As I know from my personal experience, there are some problems with vide processing:

it is very difficult to get timing correct, there are so often that some wierd black frames or flash appears at the start/end of final video, or the movie got ended too early like author wrote on github.

For example, you probably see this error when writing video lots of times:

Problem appending pixel buffer at time:XXX*

and final video got some freeze frames.This however, from my experience, it relates to two same frametime got written. To verify this, you could write little extra debug code - array of frame time that got written and print all of them.

memory handling is very flaky - it is very often you got some memory warnings or crashes that are super frustrating.
error messages are just so raw and low level that are difficult to understand.

New Approach

So I decide to try writing a simple video-processing(more specific chroma key) project using AVFoundation and OpenGL ES to see how hard it is. As you may know, the hard thing about it is you need to know OpenGL ES which is kinda scarying at the beginning, let alone combine with multithreading to acheive better performance.

I did some performance benchmark, which is not that strictly but roughly accurate. I tested against sample videos with 640x640 pixels, 8 seconds duration.

Followings is the result:

######Two Videoes Chroma Key Merging (two vidoes)

	GPUImage	AVFoundation+OpenGL ES
iPhone4	44.79s	11.90s
iPod5	4.68s	3.87s
iPhone6	1.44s	1.11s

######Three Videoes Alpha Channel Merging(three vidoes)

	GPUImage	AVFoundation+OpenGL ES
iPhone4	31.79s	9.10s
iPod5	3.71s	3.41s
iPhone6	1.62s	1.13s

######Note:

on iPhone4: GPUImage throw tons of ‘Couldn’t write a frame’ error, not be able to write a video or written video is chaotic or contains lots of feezing frames; well for custom AVFoundation+OpenGL ES,perfect full length, no frame dropping, frame synced perfectly.
on iPod5/iPhone6: final video from GPUImage ends earlier about 0.5s before full length) also frames is not fully synced, well, for custom AVFoundation+OpenGL ES, the final duration is perfect without any frame feezing.

I haven’t got much time on testing on other devices but the results looks quite promoising, plus I haven’t got any crash at all after running it many times - which GPUImage sometimes just crashes.

The problem with frame timing in GPUImage library lies in how it works. It use multithreading to get better performance, but it doens’t have a good mechanism to handle frame time synching.

Here are some logs from console:

2014-12-29 19:12:03.115 BBB[4989:60b] GPUImageMovie: captured_video_8A08021A-0886-4399-BC75-EE15C46D5963.mp4, duration: {354565/44100 = 8.040}
2014-12-29 19:12:03.126 BBB[4989:60b] GPUImageMovie: APP_Likes.mp4, duration: {723840/90000 = 8.043}
2014-12-29 19:12:03.225 BBB[4989:dd0f] ---captured_video_8A08021A-0886-4399-BC75-EE15C46D5963.mp4 newFrameReadyAtTime: 0.00
2014-12-29 19:12:03.304 BBB[4989:dd0f] ---captured_video_8A08021A-0886-4399-BC75-EE15C46D5963.mp4 newFrameReadyAtTime: 0.04
2014-12-29 19:12:03.320 BBB[4989:dd0f] ---captured_video_8A08021A-0886-4399-BC75-EE15C46D5963.mp4 newFrameReadyAtTime: 0.08
2014-12-29 19:12:03.337 BBB[4989:de03] ---APP_Likes.mp4 newFrameReadyAtTime: 0.04
2014-12-29 19:12:03.421 BBB[4989:dd0f] ---captured_video_8A08021A-0886-4399-BC75-EE15C46D5963.mp4 newFrameReadyAtTime: 0.13
2014-12-29 19:12:03.443 BBB[4989:bb03] Wrote a video frame: 0.08
2014-12-29 19:12:03.467 BBB[4989:dd0f] ---captured_video_8A08021A-0886-4399-BC75-EE15C46D5963.mp4 newFrameReadyAtTime: 0.17
2014-12-29 19:12:03.481 BBB[4989:de03] ---APP_Likes.mp4 newFrameReadyAtTime: 0.08
2014-12-29 19:12:03.542 BBB[4989:bb03] Wrote a video frame: 0.17

As you see the frametime is not perfectly synchronized between two readers, situation got even worse if you are doing three videoes merging. Why it happens? it is because how GPUImage library use multithreading.if you take a look at code in GPUImageMovie(which is the one reading frame and upload to GPU), what runSynchronouslyOnVideoProcessingQueue refers to is just dispatch the reading/uploading block to a single serial queue in GPUImageContext(singleton class to maintain one serial queue for multiple readers).

Okay, this is how frame time works in video reader, how it handles frametime passed into video writer? There is the newFrameReadyAtTime in GPUImageTwoInputFilter class, which basically says:

“only if I got both uploaded output texture in GPU from video readers, then I will pass current condition-met frametime to video writer and video writer will start writing. And any other case, I will just simply ignore any output from readers”.

To better illustrate that, if you look at the above logs, so avasset reader finished frame reading and uploading at frame time 0.00, then it simply pass it to GPUImageTwoInputFilter whom won’t start write because it hasn’t received second frame output yet, so it simply skips this frame(output is discarded) instead of pausing captured_video and starting APP_Likes because all video readers share same serial queue.

The new approach in this experiment is instead of what GPUImage currently does:

letting video readers take initiative, pass read&uploaded frame output to writer and just simply continue next frame processing
writer just simply discard whatever frames before both frame are received, which caues of inter-reader frame time out of sync

what we are gonna do is:

video writer take the initiative, reader is passive. Writer ask all readers to read/upload next frame, and it won’t start write or next-round reading at all, unless all readers are returned with its final output texture
video writer will dispatch all reader’s reading process to concurrent queue to achieve better performance instead of serial queue. We will leverage a greate feature GCD dispatch_async group to make sure all paralle-running blocks are all done before start writing.

This approache will make sure we got frames synced perfectly also gain best performance.

Breakdown

The goal that we wanna acheive is broken into three steps:

One video source: take out green color component, and write to disk (single thread, simple glsl)
Two video sources: blend using chroma key, and write to disk (multi thread, little bit complicted glsl)
Three video sources: blend using white-alpha channel, and write to disk (multi thread, simple glsl)

Let’s start from easiest first step: using avassetreader in avfoundation to read samples/frames from source video, then we upload to GPU with our magic glsl code to take out the green color component from it, after that we grab the output from fully rendered pixels in GPU, and write the transformed frames to final output movie in disk using avassetwriter avfoundation.

Key Concept Walkthrough: OpenGL ES

The difficult part in first step is OpenGL ES. You need to have basic knowledge about it, but it won’t that difficult to pick up. You need to step back and change your mind little bit to old procedure programming.

To begin with OpenGL ES, I strongly recommend Ray Wenderlich’s article OpenGL Tutorial for iOS: OpenGL ES 2.0, it has a great walkthrough lots of important concept that we’re gonna use in this series of blog. Just pay attention to how to create shader, compile shader, create vertex buffer, passing parameters and render. Note : you dont’ need to know transform matrix as video merging is plainly just two dimension.

But this is not enough yet, we need some way to upload image to GPU, which in OpenGL ES terms, called “texture”, which is well covered in second part of blog: OpenGL ES 2.0 for iPhone Tutorial Part 2: Textures. Note: you don’t need know to depth buffer, just need to know how to map texture coordinates and send pixel data to GPU.

Key Concept Walkthrough: AVFoundation

AVFoundation is what we’re gonna use for read/write frames from/to video. Apple has really good sessions from WWDC, which you could take a look. Also it has a greate resource in dev library in Export in AV Foundation Programming Guide, which covers reading/writing assets.

Key Concept Walkthrough: OpenGL ES && AVFoundation

How do combine those two keys together? I would really suggest a well reading through Apple’s sample code GLCameraRipple, what it does is:

This sample demonstrates how to use the AVFoundation framework to capture YUV frames from the camera and process them using shaders in OpenGL ES 2.0. CVOpenGLESTextureCache, which is new to iOS 5.0, is used to provide optimal performance when using the AVCaptureOutput as an OpenGL texture. In addition, a ripple effect is applied by modifying the texture coordinates of a densely tessellated quad.

After that, you need be careful as we also need use avassetwrite to write to final video, which uses AVAssetWriterInputPixelBufferAdaptor to append pixels that pulls from GPU ram. That requires a proper setup on avassetwriter.

As you see, it is not that complicted or scarying like what it says in above rage comic. We only need a fairly simple understanding of OpenGL ES(because we’re working one 2D), multithreading(because we only need to know serial/concurrent queue and dispatch_async_group), AVFoundatation(well, just setup and read/write frames).

Without further ado, let’s get our’s dirty to get our first milestone done :)

Pixel Perfect Chroma Key Blend Using GPUImage

2014-12-19T11:07:19+00:00

If you ‘re doing chroma key blend with GPUImageChromaKeyBlendFilter from GPUImage library like what I did in this post GPUImage Movie Writer Exposure Problem, then you’re very likely to find that it is very tricky to get pixel-perfect blend result:(e.g. annoying green lines around edges no matter how hard you tweak the threshold and smoothing parameter, also you need ask animation creator to be careful to not picking any close colors against the background green color which is super frustrating to them).

The reason is that if you look into the OpenGL ES GLSL code in GPUImageChromaKeyBlendFilter.m, you could find that actually it masks out the color which is close to background color(like green color by default). And this masking part is not very precise when it comes to some colors between background and character.

Then you probably wonder how Action Movie FX got this pixel-perfect blending video. After some hack, I got a clear idea about it. Following is the diagram:

+

=

As you see, it is not using green color as reference for cutting color, instead it got from a alpha-channel video (black-and-white).

Already someone posts an question for it on GPUImage issues list :Chroma or alpha on videos? #255, Brad has given some suggestions on it but there wasn’t any code about it. However, very obviously that instead of use two input filters like GPUImageChromaKeyBlendFilter, this has to use three input filters like GPUImageThreeInputFilter.m. If you looks at the code inside GPUImageChromaKeyBlendFilter, it notice that it doesnt’ have custom fragment shader to it, yes, it is designed to be subclassed to implement custom rendering.

After spent some time on tweaking it, I finally got it work. If you use GPUImage as pod dependency, I suggest to git clone a copy to your local computer, then change Podfile in XCode spike project to refer to local copy, so that you could easily debug it.

Podfile

platform :ios, '7.0'	
# ignore all warnings from all pods
inhibit_all_warnings!

#pod 'GPUImage', :path => ':git => 'https://github.com/XXX/GPUImage.git''
pod 'GPUImage', :path => '~/Documents/git/spikes/GPUImage'

Followings are the steps to do it:

####1. Open GPUImage project and create THMovieAlphaBlendFilter

 

#import "GPUImageThreeInputFilter.h"
 
@interface THMovieAlphaBlendFilter : GPUImageThreeInputFilter {
 
}

@end

####2. Change THMovieAlphaBlendFilter.m file

 
#import "THMovieAlphaBlendFilter.h"

NSString *const kTHMovieAlphaBlendFragmentShaderString = SHADER_STRING
(
 precision highp float;
 
 varying highp vec2 textureCoordinate;
 varying highp vec2 textureCoordinate2;
 varying highp vec2 textureCoordinate3;
 
 uniform sampler2D inputImageTexture;
 uniform sampler2D inputImageTexture2;
 uniform sampler2D inputImageTexture3;
 
 void main()
 {
     vec4 textureColorAlpha = texture2D(inputImageTexture, textureCoordinate);//alpha
     vec4 textureColorFX = texture2D(inputImageTexture2, textureCoordinate2); //fx
     vec4 textureColorSrc = texture2D(inputImageTexture3, textureCoordinate3); //src
 
     gl_FragColor = mix(textureColorFX, textureColorSrc, 1.0 -textureColorAlpha.r);	     
     
 });


@implementation THMovieAlphaBlendFilter
	
- (id)init;
{
    if (!(self = [super initWithFragmentShaderFromString:kTHMovieAlphaBlendFragmentShaderString]))
    {
        return nil;
    }	    	    
    return self;
}
@end

The key part is gl_FragColor = mix(textureColorFX, textureColorSrc, 1.0 -textureColorAlpha.r); , as you see the third parameter (blend value) is the value of color component from alpha-channel video.

####3. Setup and run demo code

 
@interface ViewController ()
@property (nonatomic, strong) GPUImageMovie *gpuMovieAlpha;
@property (nonatomic, strong) GPUImageMovie *gpuMovieFX;
@property (nonatomic, strong) GPUImageMovie *gpuMovieSource;
@property (nonatomic, strong) GPUImageMovieWriter *movieWriter;
@property (nonatomic, strong) THMovieAlphaBlendFilter *filter;
@end

@implementation ViewController

- (void)viewDidAppear:(BOOL)animated {
    [super viewDidAppear:animated];
    
    NSURL *urlApha = [[NSBundle mainBundle] URLForResource:@"fireworks_alpha_sd" withExtension:@"mp4"];
    self.gpuMovieAlpha = [[GPUImageMovie alloc] initWithURL:urlApha];

    NSURL *urlFX = [[NSBundle mainBundle] URLForResource:@"fireworks_sd" withExtension:@"mp4"];
    self.gpuMovieFX = [[GPUImageMovie alloc] initWithURL:urlFX];

    
    NSURL *urlSource = [[NSBundle mainBundle] URLForResource:@"source" withExtension:@"mov"];
    self.gpuMovieSource = [[GPUImageMovie alloc] initWithURL:urlSource];
    
    self.filter = [[THMovieAlphaBlendFilter alloc] init];
    
    [self.gpuMovieAlpha addTarget:self.filter];
    [self.gpuMovieFX addTarget:self.filter];
    [self.gpuMovieSource addTarget:self.filter];
    
    NSString *pathToMovie = [NSHomeDirectory() stringByAppendingPathComponent:
                             [NSString stringWithFormat:@"Documents/%@.%@",@"output", @"mp4"]];
    unlink([pathToMovie UTF8String]);
    NSURL *outputPath = [NSURL fileURLWithPath:pathToMovie];
    self.movieWriter = [[GPUImageMovieWriter alloc] initWithMovieURL:outputPath size:CGSizeMake(640.0, 640.0)];
    [self.filter addTarget:self.movieWriter];
    
    [self.movieWriter startRecording];
    [self.gpuMovieAlpha startProcessing];
    [self.gpuMovieFX startProcessing];
    [self.gpuMovieSource startProcessing];
 
    __weak typeof(self) weakSelf = self;
    [self.movieWriter setCompletionBlock:^{
        [weakSelf.gpuMovieAlpha endProcessing];
        [weakSelf.gpuMovieFX endProcessing];
        [weakSelf.gpuMovieSource endProcessing];
        [weakSelf.movieWriter finishRecording];
    }];
    
}
@end

Extreme care should be taken for the order of addTarget:self.filter part, as this order maps strictly to the order in GLSL fragment shader.

There you go, you should be able to get pixel-perfect blended videos, no pulling hairs from animation creators.

All videos(including alpha-channel, fx, source) mentioned in above code could be found here : sample videos. Just quick reminder, the sample vidoeos are just for studying code purpose, do not distribute or use for commercial.

在Parallels Desktop 9运行Windows 8 Phone模拟器

2014-07-19T04:17:52+00:00

在Parallels Desktop 9中跑起来Windows Phone 8的模拟器来进行开发，准备工作大概可以分为几部分：

操作系统

Windows 8 64位 Pro版本的iso

开发工具

Microsoft Visual Studio Express 2012 for Windows 8 选择下载VS2012_WINEXP_enu.ISO就好

##Windows Phone SDK 8.0

建议直接下载iso文件，当然英文最好: Windows Phone SDK 8.0 iso

##安装顺序 windows 8 -> visual studio -> wp8 sdk

##Parallel Desktop 9

具体安装流程可以参考使用 Parallels Desktop 在 Mac 上安装 Windows 和开发工具这里有一点需要注意的是，在[步骤 2：设置 Parallels Desktop]中第六步之后，你还需要选中设置面板中第二个tab[Options],下面的左侧的[Optimization]，右侧内容中[Enable nested virtualization]。

在系统安装完毕之后，我们需要打开Hyper-V功能，这是能让模拟器能跑起来的重要条件。

依次打开 Control Panel -> Programs and Features -> Turn Windows features on or off -> tick Hyper-V

然后重启电脑。

接下来就是用ultraiso依次加载和安装visual studio和windows phone 8 sdk。

到wp8 sdk安装完毕之后，你应该能看到如下的界面，那就表示安装成功，你可以直接打开vs跑起来了。

接下来就简单了，跑起来Hello World.

References

Unity 2D Gradient Background Transition

2014-06-21T05:31:22+00:00

When develop 2D games in Unity, sometime we want to make some transition on day-night cycle(e.g. sky color change)

So I have one design for start state of dawn like following:

(bottom color: fe805e, top: 527fc1)

and the end state of dawn:

(bottom color: fa8856, top: 7ae0ec)

The final effect should be like this:

Graident

To achieve the gradient background, we need to apply a new gradient shader. The shader would accept one main texture, and two colors.

 
Shader "Custom/Gradient" {
Properties {
	[PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
    _Color ("Bottom Color", Color) = (1,1,1,1)
    _Color2 ("Top Color", Color) = (1,1,1,1)
    _Scale ("Scale", Float) = 1
}
 
SubShader {
    Tags {"Queue"="Background"  "IgnoreProjector"="True"}
    LOD 100
 
    ZWrite On
 
    Pass {
        CGPROGRAM
        #pragma vertex vert  
        #pragma fragment frag
        #include "UnityCG.cginc"
 
        fixed4 _Color;
        fixed4 _Color2;
        fixed  _Scale;
 
        struct v2f {
            float4 pos : SV_POSITION;
            fixed4 col : COLOR;
        };
 
        v2f vert (appdata_full v)
        {
            v2f o;
            o.pos = mul (UNITY_MATRIX_MVP, v.vertex);
            o.col = lerp(_Color,_Color2, v.texcoord.y );
//            o.col = half4( v.vertex.y, 0, 0, 1);
            return o;
        }
       
 
        float4 frag (v2f i) : COLOR {
            float4 c = i.col;
            c.a = 1;
            return c;
        }
            ENDCG
        }
    }
}

Then create a material with that shader and assign the mat to sprite render, finally in script:

 
sRender.material.SetColor("_Color", Color.black); // the bottom color
sRender.material.SetColor("_Color2", Color.white); // the top color

Transition

 

private float duration = 1.0f;
	public SpriteRenderer sRender;
	// Update is called once per frame
	Color HexToColor(string hex)
	{
		byte r = byte.Parse(hex.Substring(0,2), System.Globalization.NumberStyles.HexNumber);
		byte g = byte.Parse(hex.Substring(2,2), System.Globalization.NumberStyles.HexNumber);
		byte b = byte.Parse(hex.Substring(4,2), System.Globalization.NumberStyles.HexNumber);
		return new Color32(r,g,b, 255);
	}


	void Update () {
		float lerp = Mathf.PingPong(Time.time, duration) / duration;
		sRender.material.SetColor("_Color", Color.Lerp(HexToColor("fe805e"),HexToColor("fa8856"),  lerp)); //bottom
		sRender.material.SetColor("_Color2", Color.Lerp(HexToColor("527fc1"),HexToColor("7ae0ec"),  lerp)); //top
	}

TPLink WR720N刷OpenWrt - 修砖

2014-05-04T09:06:50+00:00

一年一度的NBA季后赛开始了，今年季后赛第一轮更是尤其精彩，连续两天五场抢七，还有惊心动魄的火箭打开拓者，看的我这个篮球迷如痴如醉。但是比赛一般都在早上，而且比赛时间还挺长的，如果看完了再去上班肯定稳稳的迟到，如果晚上回来想看录像了又觉得网速那个点又慢，而且我对国内的解说说实话也不是感冒，看英文的可能还舒服点，没那么吵吵。所以我希望在比赛结束了（一般都是中午左右），在上班的间隙我把种子下好，然后上传到比如我家里路由器暴露在外面上某个网址上，可以利用迅雷下载的速度优势，在下班回家前就把高清录像下好了。我回到家里的时候，直接从电脑上stream路由器的存储媒介（通常是硬盘）上下好的视频。

选路由器是因为我不可能把家里的笔记本一直开一天吧，这个也太浪费了把。而且路由器本来就是在那开一天的，倒也不浪费。所以怎么捯饬这玩意可以达到我上面美好的愿望了？就是OpenWrt了。简单的说，我将要干的事情就是把路由器出厂自带的固件也换成Openwrt。我发现网上有不少关于这方面的教程（大部分还是703N，有点老了），但是没有讲的很清楚从头到尾很直白的那种，这篇博客就是记录自己刷机的过程

##首次刷机尝试

手头有的东西名目:

* 一个很早之前买的TPLink WR720N迷你路由
* 一根microUSB线
* 网线

那选择WR720N也是因为首先它便宜啊，而且它自带一个USB口(方便挂载外设），一个microUSB口，一个LAN口，还有一个WAN/LAN口，真实麻雀虽小，五脏俱全。

而且Openwrt官方已经添加了对WR720N专有固件的支持(你可以找到openwrt支持的所有硬件列表Table of Hardware - Router type)，使得刷机愈加容易。你需要做的就是去这个地址http://downloads.openwrt.org/snapshots/trunk/ar71xx/找到openwrt-ar71xx-generic-tl-wr720n-v3-squashfs-factory.bin和openwrt-ar71xx-generic-tl-wr720n-v3-jffs2-sysupgrade.bin，然后下载到本地。

接下来就是登陆路由器,720N就是192.168.1.253,然后找到左侧最下面的升级，选取将本地刚刚下好的factory.bin(不是sysupgrade.bin)）升级,耐心等重启之后就可以了。

为什么这里是factory而不是sysupgrade了？ FAQ before installing OpenWrt

What is the difference between the different image formats?

a factory image is one built for the bootloader flasher or stock software flasher

sysupgrade image (previously named trx image) is designed to be flashed from within openwrt itself

##刷成砖头了是的，应该很简单，应该木有问题的，但是很不幸我的刷成了砖头了，ping不通，IP地址无效，重设了N遍都木有任何反应。:scream:

怎么办了？我大概了解下路由器启动过程，一般路由器都有一个Flash卡（相当于电脑的硬盘，里面有你个操作系统文件等等），里面有你写好或者下好的固件.当路由器启动时，有一个启动引导程序uboot（相当于windows上BIOS),一般来收它会从Flash卡读取固件（相当于windows比如默认从硬盘上启动操作系统）并加载到内存中。

这里就是一般来说就是Flash卡中的固件已经损坏了，我们需要擦除里面已有固件信息，然后将原厂固件刷进去。但是这个怎么操作了，我们都进不去路由？

##TTL修砖除了砸了之外还是有办法挽救的 - TTL线。我也不太清楚这个具体原理，大概就是你需要找到路由器板子上的两个接口TP_IN和TP_OUT(可以用来接受和发送指令),这里相当于外设，然后与电脑(COM)进行串口通信。

但是笔记本木有COM端口了，需要将USB转换成COM端口，然后与路由器电路板上连接，接着使用串口通信的软件设置对应的串口号以及波特率，这样就可以发送指令到路由器电路板，从而修复固件。

我们需要的东西:

* USB转串口的TTL线, 可以从淘宝上买[USB转TTL模块](http://item.taobao.com/item.htm?_u=tc8grg404cf&id=14126761542)）
* 电烙铁 
* 杜邦线（用于焊接和连接USB模块）

就可以了。

####1. 接线接下来就是拆机找到TP_IN和TP_OUT两个引脚。这个拆机还真有点技巧性。因为720N这个塑料外壳还真是一次性的，用蛮力的话盒子就没法用了，那么就不能插电源充电了，就只能是用microusb线充电了。但是因为引脚实在板子背面所以我们必须要将板子取出来。我想了一个办法就是拿烧红的刀直接去砍网口那边的两侧，这样一来板子可以取出来而且以后要是弄好了我还可以放回去，插电源充电也木有问题。

成功拆机了。

如果你仔细看背面中间，就可以发现TP_IN和TP_OUT连个引脚，但是它们都非常小，这就使得焊接非常难。使用电烙铁小心将杜邦线焊接到这两个接口上，另外一根线作为接地线GND随意焊接到哪都行，图方便的话直接焊接到USB接口外壳上也可以。

最后将引出的杜邦线连接到USB转TTL模块上。这里USB模块上写着黑GND, 白RXD,绿TXD.我们要做就是将引出的TP_IN针连接到白RXD上， TP_OUT连接到绿TXD口上。

注意注意：千万别将TP_IN或者TP_OUT连接到VCC口上，不然直接就把板子给烧了。（我去，我最后不小心看花眼连接错了直接把板子给烧了，是的，程序员不能在困的时候干正事）

####2. 安装驱动你直接连接好线是不能直接用的，串口还是无法被识别出来。你需要下载对应的驱动，也就是PL2303 USB to TTL驱动， mac版以及windows版.

给路由器供电，这里我们使用microUSB来供电，接下来就是去设备管理器中查看端口(COM和PT)下面有没有东西，如果有类似于Prolific USB-to-Serial Comm port(COM4)，就表示识别成功。然后点击属性将其波特率改成115200.

####3. 调试到这一步就差不多可以干正事了，我们可以使用串口通信软件来发送指令来刷固件，这里用的是SecureCRT。很简单，我们创建一个串口连接，将其COM口设成设备管理器中显示的端口号，波特率设为115200.

拔掉microUSB线，然后重新连上，你就可以看到SecureCRT中开始滚动了，你应该可以看到如下信息，没有反应的话，检查你得rx/tx是否接反了。

是的，我们看到了uboot的引导信息。接下来就是怎么样打断其正常引导过程，因为Flash卡中的固件已经损坏了。但是我们不可能把Flash卡拔下来，把原厂库件刷进入，然后插回去重启。我们就需要告诉uboot从其他地方这里比如就是我们的电脑下载到flash卡中。这里就可以配合tftp来实现从电脑上下载。为什么是tftp了？如果你后面进入了uboot输入模式，输入help就可以看到其中有一个选项：

tftpboot- boot image via network using TFTP protocol

可以理解为TFTP是uboot能理解的轻量级FTP.

用网线一端连接路由器的WAN/LAN口，一端连接电脑。打开电脑,设置ip地址：

IP: 192.168.1.3
Mask: 255.255.255.0
gateway/router: 空白 不要填 

然后重启路由，在SecureCRT中看到如下信息的时候

U-Boot 1.1.4 (Jun 20 2012 - 17:03:09)

AP121 (ar9330) U-boot

DRAM:  32 MB
led turning on for 1s...
id read 0x100000ff
flash size 4194304, sector count = 64
Flash:  4 MB
Using default environment
...

迅速在键盘上敲下tpl,要快，相当于电脑上按F12来进入BIOS. 这个时候串口会停止打印, 并出现hornet>,就表示进入了输入模式。

在输入命令之前，回到电脑上下载安装好TFTP软件, 然后选择共享文件夹为比如桌面/openwrt，在这个文件夹下放入下载好的原厂固件wr720nv3。

回到SecureCRT中，按照如下步骤输入命令：

hornet> setenv serverip 192.168.1.3

这里设置TFTP服务器的地址，这样wr720nv3.bin可以被uboot访问到

hornet> tftpboot 0x80000000 wr720nv3.bin

这里wr720nv3就是tftp共享的文件名字,确保TFTP服务器打开状态你应该可以看到如下输出：

eth1 link down
Using eth0 device
TFTP from server 192.168.1.3; our IP address is 192.168.1.111
Filename '720.bin'.
Load address: 0x80000000
Loading: *.#################################################################
. #################################################################
. #################################################################
. #################################################################
. #################################################################
. #################################################################
. #################################################################
. #################################################################
. #################################################################
. #################################################################
. #################################################################
. ######################################################
done
Bytes transferred = 3932164 (3c0004 hex) 

hornet>erase 0x9f020000 +0x3c0000

 First 0x2 last 0x3d sector size 0x10000
 ....   2....   3....   4....   5....   6....   7....   8....   9....  10....  11....  12....  13....  14....  15....  16....  17....  18....  19....  20....  21....  22....  23....  24....  25....  26....  27....  28....  29....  30....  31....  32....  33....  34....  35....  36....  37....  38....  39....  40....  41....  42....  43....  44....  45....  46....  47....  48....  49....  50....  51....  52....  53....  54....  55....  56....  57....  58....  59....  60....  61
 Erased 60 sectors

hornet> cp.b 0×80000000 0x9f020000 0x3c0000

 Copy to Flash... write addr: 9f020000
 done

hornet>bootm 0x9f020000

完整的命令是如下：

setenv serverip 192.168.1.3
tftpboot 0x80000000 wr720nv3.bin
erase 0x9f020000 +0x3c0000
cp.b 0x80000000 0x9f020000 0x3c0000
bootm 0x9f020000

基本上刷机完成。

####4. 验证

怎么验证是否成功了？重启路由，插入网线，连接无线网络（好像是TP-Link-MXXXX不记得了），输入192.168.1.253。

登进去就说明没什么问题了，修好了, 修砖完成。

#####引用

UrbanAirship Android Received push with invalid authorization on platform GCM

2014-03-19T12:46:47+00:00

I was trying integrate Urban Airship to an android application so that I could send push notification to all registered android devices via Google’s GCM (Google Cloud Messaging for Android) service.

Urban Airship does have a really good documentation about to how to set it up on Android: Android: Getting Started with Push. So I followed the instruction above and integrated with android app locally. But when I try to test the push notification by going to Messages -> Test Push in the left panel like following screenshot .

It keeps saying Received push with invalid authorization on platform GCM.

##Problem breakdown

I searched in google and found several same issues reported by people, e.g., Received push with invalid authorization on platform GCM in Urban push console and Received push with invalid authorization on platform GCM. The thing is that it works fine on iOS but not on Android, the kind-hearted guy from Urban Airship suggested to look through their GCM Troubleshooting Guide.

Well, I did verify each steps illustrated in the troubleshooting guide, the api key, package, sender id and so on. Still no clue for it.I take a step back and think the problem clearly refers to “invalid authroization on platform GCM”, then it should be something wrong with GCM settings.

Here is the settings in google api console:

There is SHA1 string and a package name for it, you have to make sure you have right SHA1 and package name from your app. Based on the documentation about Keys, access, security, and identity. I need doublec check the apk generated should have same SHA1 and package name. To verify the SHA1 string, there is post in stackoverflow How to find out which key was used to sign an app?

2:35 ~/Desktop/frames/PushSample-debug-unaligned/META-INF $ keytool -printcert -file CERT.RSA
Owner: CN=Android Debug, O=Android, C=US
Issuer: CN=Android Debug, O=Android, C=US
Serial number: 517787d8
Valid from: Wed Apr 24 15:20:56 CST 2013 until: Fri Apr 17 15:20:56 CST 2043
Certificate fingerprints:
	 MD5:  9C:DC:4C:FA:5A:74:F0:83:9D:17:D8:3B:1A:C3:BD:E5
	 SHA1: 34:10:4F:F9:9D:63:7A:56:BE:94:AD:07:B3:80:3B:A1:EE:9A:E6:67
	 Signature algorithm name: SHA1withRSA
	 Version: 3

and compare with the SHA1 string in ~/.android/debug.keystore:

12:36 ~/Desktop/frames/PushSample-debug-unaligned/META-INF $ keytool -list -keystore ~/.android/debug.keystore
Enter keystore password:android	
Keystore type: JKS
Keystore provider: SUN	
Your keystore contains 1 entry	
androiddebugkey, Apr 24, 2013, PrivateKeyEntry,
Certificate fingerprint (MD5): 9C:DC:4C:FA:5A:74:F0:83:9D:17:D8:3B:1A:C3:BD:E5

The SHA1 string is the same, then how about the package name?

Refer to this post Resolving the package name of Android APK, I’m able to print the apk infos about package name:

12:40 ~/Desktop/frames $ /Users/tuo/Documents/SDKS/adt-bundle-mac-x86_64-20131030/sdk/build-tools/android-4.4/aapt  dump badging PushSample-debug-unaligned.apk
package: name='com.reigndesign.rdapp1' versionCode='1' versionName='1.0'
sdkVersion:'7'
targetSdkVersion:'19'
uses-permission:'android.permission.INTERNET'
...

Also the package name matches.

Here is the question the package name and SHA1 are correct, then how come it still says “invalid authorization” ?

##Android key vs Server key

My instinct that accumulated from countless red-eye days of programming told me that I should rollback everything and start walk through the documentation as carefully as I can from scratch.

And I did find out why.

In one section about Get your API Key from Google, it says:

4.Generate an API Key	
	A. Click on the text where it says “Google Cloud Messaging for Android” in the image above.		
	B. This takes your to the Google APIs page. Click on API Access.		
	C. Urban Airship takes care of API Access authorization for you, so you do not need to create an OAuth 2.0 client ID.		
	D. Click on “Create a new Server key...” to generate your API Key.		
	E. Do not specify any IP addresses in the form, and click “Create”

What I realized is that holy **, it is god dammned Server key instead of Android key. I did go back and take a detail look at what android key is about:

Create an Android key and configure allowed Android applications
	This key can be deployed in your Android application. API requests are sent directly to Google from your client Android device. Google verifies that each request originates from an Android application that matches one of the certificate SHA1 fingerprints and package names listed below. You can discover the SHA1 fingerprint of your developer certificate using the following command:
	keytool -list -v -keystore mystore.keystore

Then I understand it better. Because ubarn airship is acting on our app’s behalf dealing with GCM platform rather than let our app do the dirty job. So serve key makes sense to me now.

So I created a Server key and try it again, no more errors.

GPUImage Photo(Still)/Video - Color Degradation/Too Much Brightness

2014-03-02T05:29:59+00:00

I have mentioned this problem before in my previous post GPUImage Movie Writer Exposure Problem.

I thought I have solved problem but still the color rendered in GPUImageView is weird - it doesn’t look that much nature. I re-examine the change and find there is some sample code called GLCameraRipple from Apple which pretty much does same thing - color conversion from YUV to RGB.

After I download and check the code in GLCameraRipple, I find that actually the shader is pretty much the same. I changed the shader string to what I mentioned in GPUImage Movie Writer Exposure Problem, tried it out but the rendered image still isn’t right.

Then I trace from shader string down to the configurations/settings of the OpenGL ES, more specifically the texture uploading.

Here are the code in GPUImageVideoCamera.m:

 
// Y-plane
glActiveTexture(GL_TEXTURE4);
if ([GPUImageContext deviceSupportsRedTextures])
{
//                err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, coreVideoTextureCache, cameraFrame, NULL, GL_TEXTURE_2D, GL_RED_EXT, bufferWidth, bufferHeight, GL_RED_EXT, GL_UNSIGNED_BYTE, 0, &luminanceTextureRef);
    err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, coreVideoTextureCache, cameraFrame, NULL, GL_TEXTURE_2D, GL_LUMINANCE, bufferWidth, bufferHeight, GL_LUMINANCE, GL_UNSIGNED_BYTE, 0, &luminanceTextureRef);
}
else
{
    err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, coreVideoTextureCache, cameraFrame, NULL, GL_TEXTURE_2D, GL_LUMINANCE, bufferWidth, bufferHeight, GL_LUMINANCE, GL_UNSIGNED_BYTE, 0, &luminanceTextureRef);
}

You would probably wonder why those two code in both branches is the same.As you see, the only difference is that GL_RED_EXT vs GL_LUMINANCE. Then I’m curious about which value is used in apple’s sample code.

Open file RippleViewController.m, scroll down then you can see the following code snippet:

 
// CVOpenGLESTextureCacheCreateTextureFromImage will create GLES texture
// optimally from CVImageBufferRef.

// Y-plane
glActiveTexture(GL_TEXTURE0);
err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, 
                                                   _videoTextureCache,
                                                   pixelBuffer,
                                                   NULL,
                                                   GL_TEXTURE_2D,
                                                   GL_RED_EXT,
                                                   _textureWidth,
                                                   _textureHeight,
                                                   GL_RED_EXT,
                                                   GL_UNSIGNED_BYTE,
                                                   0,
                                                   &_lumaTexture);
if (err) 
{
    NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
}

So the shader string is much the same, well the texture uploading settings is different. I tried to change all the settings in GPUImageVideoCamera.m and GPUImageMovie.m.

Now run the app again, compare the image with previous one, this time it looks fine.

The complete code change is in this commit: fix color degradation.

GPUImage Merge Videos with Chroma Key - GPUImageMovieWriter Two Audio Tracks

2014-02-16T01:38:51+00:00

Let’s continue second topic about Merging videos with GPUImage: Writing both audio tracks of source videos to final output video.

 
...

self.gpuMovieA = [[GPUImageMovie alloc] initWithURL:..];
self.gpuMovieFX = [[GPUImageMovie alloc] initWithURL:...];

...

self.movieWriter.shouldPassthroughAudio = YES;
self.gpuMovieA.audioEncodingTarget = self.movieWriter;
[self.gpuMovieA enableSynchronizedEncodingUsingMovieWriter:self.movieWriter];

...

So you want to write audio track of one of two source video to destination video, above code would works quite well.

For example, in above code we want to have audio of gpuMovieA video show up in final video. Then sometimes we found that we need to merge two audio tracks in each source video into final output video.

There are some issues in github about audio writing : Audio writing issues with GPUImageMovieWriter and Merging of video and audio.

Someone in the issue mentioned with frustration:

I gave up and wrote my own code. GPUImage is great but it has so many issues and the code is not very easy to read for openGL noobs like me. I learned more by writing it myself - all roads lead to rome, and all of GPUImage is basically based on a few examples out there on the net anyhow..

Well, Brad Larson has just commented to that request:

There currently is no way to do this kind of audio mixing. Only one audio source is used at a time.

#Problem Breakdown

Merging video tracks does work but audio doens’t is because the way how it works. For video tracks merging, there are two GPUImageMovie instances and each starts reading frames, uploading image buffer to GPU then grabbing the reference to rendered texture, passing to GPUImageMovieWriter instance. Then GPUImageMovieWriter bind references of two textures to chroma key shader’s input sample textures, rendered it. Finally grabbing the rendered output texture and write it to output video.

However audio tracks works quite differently, as you couldn’t mix two CMSampleBufferRef of audio track output. How about in the GPUImageMovieWriter we use AVMutableComposition to mix two audio tracks and output it as a source for asset writer to write it to final output video.

 
- (void)startMixAudio{
     if(self.rawVideoPath){
         //start reading
         NSURL *audioURL = [NSURL fileURLWithPath:self.rawVideoPath];
         NSURL *audioURL2 = [[NSBundle mainBundle] URLForResource:@"BOOMi_Greet" withExtension:@"mp4"];
 
         AVURLAsset* audioAsset = [[AVURLAsset alloc]initWithURL:audioURL options:nil];
         AVURLAsset* audioAsset2 = [[AVURLAsset alloc]initWithURL:audioURL2 options:nil];
 
 
         AVMutableComposition* mixComposition = [AVMutableComposition composition];
 
         AVMutableCompositionTrack *compositionCommentaryTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeAudio
                                                                                             preferredTrackID:kCMPersistentTrackID_Invalid];
         [compositionCommentaryTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, audioAsset.duration)
                                             ofTrack:[[audioAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0]
                                              atTime:kCMTimeZero error:nil];
 
 
         AVMutableCompositionTrack *compositionAudioTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeAudio
                                                                                        preferredTrackID:kCMPersistentTrackID_Invalid];
         [compositionAudioTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, audioAsset.duration)
                                        ofTrack:[[audioAsset2 tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0]
                                         atTime:kCMTimeZero error:nil];
 
 
 
 
         // --- video reader
 
         AVAssetReader *assetReader = [AVAssetReader assetReaderWithAsset:mixComposition error:nil];
 
 
         AVAssetReaderAudioMixOutput * assetReaderTrackOutput =
                 [[AVAssetReaderAudioMixOutput alloc] initWithAudioTracks:[mixComposition tracksWithMediaType:AVMediaTypeAudio]
                                                            audioSettings:nil];
 
 
 //        AVAssetTrack *assetTrack = [[mixComposition tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];
 //        AVAssetReaderTrackOutput *assetReaderTrackOutput = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:assetTrack outputSettings:NULL];
         [assetReader addOutput:assetReaderTrackOutput];
         [assetReader startReading];
 
 
         [assetWriterAudioInput requestMediaDataWhenReadyOnQueue:movieWritingQueue usingBlock:^
         {
             NSLog(@"Asset Writer ready :%d", assetWriterAudioInput.readyForMoreMediaData);
 
             while (assetWriterAudioInput.readyForMoreMediaData & !audioEncodingIsFinished)
             {
                 CMSampleBufferRef nextBuffer;
 
                 if([assetReader status] == AVAssetReaderStatusReading && (nextBuffer = [assetReaderTrackOutput copyNextSampleBuffer]))
                 {
                     if (nextBuffer)
                     {
                         NSLog(@"append buffer NextBuffer");
                         [assetWriterAudioInput appendSampleBuffer:nextBuffer];
                     }
                 }
 
                 else
                 {
                     NSLog(@"audio wrint done");
                     audioEncodingIsFinished = YES;
                     [assetWriterAudioInput markAsFinished];
                 }
             }
         }];
     }
 
  }

To see more details on this , you can check my commit on github: Writing Two Audio Tracks

GPUImage Merge Videos with Chroma Key - GPUImageMovie Audio Playback

2014-02-15T01:38:51+00:00

Well, using GPUImage to merge two videos with chroma key is quite easy as what we see in following code snippet:

 
NSURL *urlA = [[NSBundle mainBundle] URLForResource:@"captured_video" withExtension:@"mov"];
self.gpuMovieA = [[GPUImageMovie alloc] initWithURL:urlA];
self.gpuMovieA.playAtActualSpeed = true;

NSURL *urlB = [[NSBundle mainBundle] URLForResource:fxMovie.movieName withExtension:@"mov"];
self.gpuMovieFX = [[GPUImageMovie alloc] initWithURL:urlB];
self.gpuMovieFX.playAtActualSpeed = true;

self.filter = [[GPUImageChromaKeyBlendFilter alloc] init];
[self.filter forceProcessingAtSize:CGSizeMake(640, 640)];

[self.gpuMovieFX addTarget:self.filter];
[self.gpuMovieA addTarget:self.filter];
[self.filter addTarget:self.outputView];

//setup writer
NSString *pathToMovie = [NSHomeDirectory() stringByAppendingPathComponent:@"Documents/output.mov"];
unlink([pathToMovie UTF8String]); // If a file already exists, AVAssetWriter won't let you record new frames, so delete the old movie
self.convertedVideoPath = [NSURL fileURLWithPath:pathToMovie];
self.movieWriter = [[GPUImageMovieWriter alloc] initWithMovieURL:self.convertedVideoPath
                                                            size:CGSizeMake(640.0, 640.0) compressVideoOutput:YES];
[self.filter addTarget:self.movieWriter];    
self.movieWriter.shouldPassthroughAudio = YES;
self.gpuMovieA.audioEncodingTarget = self.movieWriter;
//add this line otherwise it will cause "Had to drop an audio frame" which cozes the saved video lose some sounds
[self.gpuMovieA enableSynchronizedEncodingUsingMovieWriter:self.movieWriter];

[self.movieWriter startRecording];
[self.gpuMovieA startProcessing];
[self.gpuMovieFX startProcessing];

__weak typeof(self) *sself = self;

[self.movieWriter setCompletionBlock:^{
    sself.gpuMovieA.audioEncodingTarget = nil;
    [sself.gpuMovieFX endProcessing];
    [sself.gpuMovieA endProcessing];
    [sself.movieWriter finishRecording];
    sself.recording = NO;
}];

Above code does work. However you probably notice that it has three problems:

GPUImageMovie doens’t have audio playback
The merged video only have one audio track (not both tracks from each video)
The merged video have some color degradation problem

So I’m walking through each items to show how to fix it. Well, I’m not sure the solution is best, at least it works(somehow #_#), hopefully it could inspire you with even better approach.

Topics would cover in this post are:

GPUImageMovie audio playback
Movie writing with two audio tracks
Fix the color degradation (too much brightness and exposure)

GPUImageMovie Audio Playback

Well, this topic has been discussed quite a lot in GPUImage issue list, e.g. Playing audio, audio in video filter issue, GPUImageMovie Audio Playback.

Someone commented after the issue saying:

I also need this feature. I’m considering adopting a simpler work around. Play the audio of the origin file of the GPUImageMovie. Probable the sync between video and audio will be a problem since GPUImageMovie may not play at the original speed.

Then it comes an approach, which is that add a basic audio player to support playback (using AVAudioPlayer). Luckily he has uploaded a gist for reference: Add Audio Playing to GPUImageMovie.

You can set one of property (i.e. playSound) in GPUImageMovie to true when you want to have audio playback.

But someone replied to that comment saying he got some either crash or slience with new approach. I tried it little bit and found you probably need hack it little bit to get it work(as the codebase evolves so quickly, quite oftenly you need to do a merge @_@)

You can refer to my commit in github to see what exactly I did: Added Play Audio support in GPUImageMovie.

Given the fact the hack is using AVAudioPlayer to play sound of the video, you must pay attention to the format of video. So if you use .mov video, you could get some errors about initializing audio player instance. Well, that is because AVAduioPlayer doesn’t support .mov format:Play Sound of .mov file with AVAudioPlayer. However, .mp4 file is okay.

Unity GUI自适应屏幕分辨率（一）布局自适应

2014-01-11T06:21:51+00:00

在用Unity做游戏开发中，碰到一个头疼的问题就是如何设计自适应屏幕分辨率的GUI菜单。因为现在的屏幕尺寸太多，分辨率各个不一样，但是我们又不想为某一种具体的屏幕分辨率做特别的处理，有木有一种通用的方案来解决这个问题了？

我个人在开发的过程中碰到了不少这方面的问题，所以下面六个话题中一一阐述这些常见的问题，算是抛砖引玉吧。

坐标矩阵变化实现布局自适应
标清高清图片资源的加载
Label高清实现（字体模糊问题）
Scroll view的矩阵变化
GUI Box vs DrawTexture
GUI 全屏背景图片

这里我们先谈第一个问题坐标矩阵变化实现布局自适应。

选取基准尺寸

通常你需要选择一个基准的屏幕尺寸，象现在开发的应用也需要跨平台在iOS(iPhone/iPad)/Android都可以运行，我这边选取的是iphone4的屏幕尺寸: 480 * 320. 设计师设计的GUI的素材时就是按照这个尺寸来设计。但是紧接着的问题是如何保证它在其他不同尺寸/分辨率的平台上运行时不会出现各种诡异的位置大小错乱了。

举一个实际的例子来更好描述这个问题，比如我们的游戏是水平方向的，然后游戏进行中间的暂停界面中，有三个角落有按钮或着标签，屏幕中间有一个按钮，如下图所示：

很简单的代码：

 
void OnGUI()
{

	GUI.Box(new Rect(15 , 10, 83, 49), bg_score);
	GUI.Box(new Rect(372, 10, 98, 44), bg_time);
	
	if (GUI.Button(new Rect(5, 280, 67, 41), bt_pause))
	{
		//pause the scene
	}

}

伸缩居中

在Unity中我们将Game窗口的模式选择为iPhone Wide(480x320)，然后运行游戏，木有什么问题。紧接着尝试运行在iPhone 4G Wide(960x640)，你就会发现问题了，整个格局错乱了，并没有有比例的伸缩，然后全部堆到了左边.

所以可以想到既然有了基准的屏幕尺寸，其他尺寸的处理必然需要针对这个基准来变化，我们需要的是在基准屏幕上各个元素都按照一定的比例放大或者缩小，水平和竖直方向的伸缩比列一定得是同步的，这样一来保证它们之间的相对位置保持不变。在Unity中GUI系统中我们就需要运用到GUI.Matrix矩阵变化。

要解决这个问题，我们可以定义一个基准尺寸，我们这里是480*320.

 
public static Vector2 NativeResolution  = new Vector2(480, 320);

然后了，我们要让长宽按照这个基准来变化，包括首先是伸缩放大或缩小，其次是在变化之后使其保持居中。

 
private static float _guiScaleFactor = -1.0f;
private static Vector3 _offset = Vector3.zero;
	
static List<Matrix4x4> stack = new List<Matrix4x4> ();
public void BeginUIResizing()
{
	Vector2 nativeSize = NativeResolution;
	
	_didResizeUI = true;
	
	stack.Add (GUI.matrix);
	Matrix4x4 m = new Matrix4x4();
	var w = (float)Screen.width;
	var h = (float)Screen.height;
	var aspect = w / h;
	var offset = Vector3.zero;
	if(aspect < (nativeSize.x/nativeSize.y)) 
	{ 
	//screen is taller
	    _guiScaleFactor = (Screen.width/nativeSize.x);
	    offset.y += (Screen.height-(nativeSize.y*guiScaleFactor))*0.5f;
	} 
	else 
	{ 
	// screen is wider
	    _guiScaleFactor = (Screen.height/nativeSize.y);
	    offset.x += (Screen.width-(nativeSize.x*guiScaleFactor))*0.5f;
	}
	
	m.SetTRS(offset,Quaternion.identity,Vector3.one*guiScaleFactor);
	GUI.matrix *= m;	
}

public void EndUIResizing()
{
	GUI.matrix = stack[stack.Count - 1];
	stack.RemoveAt (stack.Count - 1);
	_didResizeUI = false;
}

紧着这我们在OnGUI方法中的开头和结尾分别调用BeginUIResizing和EndUIResizing来变化矩阵。

 
void OnGUI()
{
	BeginUIResizing(); //call this in the beginning of method

	GUI.Box(new Rect(15 , 10, 83, 49), bg_score);
	GUI.Box(new Rect(372, 10, 98, 44), bg_time);
	
	if (GUI.Button(new Rect(5, 280, 67, 41), bt_pause))
	{
		//pause the scene
	}

	EndUIResizing(); //call this in the end of method
}

这里我们根据长宽比，算出伸缩比例，然后为了保证伸缩之后的画面能始终在屏幕中间，我们要算出多出来偏移量，最后我们根据这个偏移量和缩放比例对矩阵进行变化。

然后我们再在分辨率为960*640的情况下运行。

iPhone4 960 * 640:

iPhone5 1136 * 640:

iPad 1024 * 768:

你可以看到在其他尺寸的屏幕上伸缩都没有问题，而且元素都居中。但是有一个问题，你发现在iPhone5和iPad上几个标签按钮的位置有点不太对，他们需要像iPhone4一样紧贴两边，而在iPhone5和iPad上却不是这样。

偏移量

要想解决这个问题的理解这个矩阵变化是如何工作的。这个偏移量是变换之后算出来的偏移量。所以要想让GUI元素在变换之后依然在保持屏幕的边缘，我们需要将x,y减去这偏移量，于是有了下面的代码:

 
void OnGUI()
{
	BeginUIResizing(); //call this in the beginning of method

	GUI.Box(new Rect(15 - offset.x/guiScaleFactor , 10 - offset.y/guiScaleFactor,
	 	83, 49), bg_score);
	GUI.Box(new Rect(372 + offset.x/guiScaleFactor, 10 - offset.y/guiScaleFactor, 
		98, 44), bg_time);
	
	if (GUI.Button(new Rect(5 - offset.x/guiScaleFactor, 280 + offset.y/guiScaleFactor, 
		67, 41), bt_pause))
	{
		//pause the scene
	}

	EndUIResizing(); //call this in the end of method
}

这里要记住这个偏移量是offset.x/guiScaleFactor, 而不是offset.x，因为这个坐标是基于基准矩阵来的。

将代码重新跑一遍：

iPhone5 1136*640

iPad 1024 * 768:

这个木有问题了。

接下来我们就需要讨论加载标清高清图片的问题。

GPUImage Movie Writer Exposure Problem

2014-01-05T07:38:24+00:00

There is updated post on this: GPUImage Photo(Still)/Video - Color Degradation/Too Much Brightness

GPUImage is a great library for processing image and video on iOS. Recently I have used it for merging and blending two videos by applying chroma key filter. The idea is we create two GPUImageMovie objects, one GPUImageChromaKeyBlendFilter and finally the GPUImageMovieWriter object. The code is like following:

 
NSURL *urlA = [NSURL fileURLWithPath:self.tempMoviePath];
self.gpuMovieA = [[GPUImageMovie alloc] initWithURL:urlA];
self.gpuMovieA.playAtActualSpeed = true;
self.gpuMovieA.playSound = true;

NSURL *urlB = [[NSBundle mainBundle] URLForResource:fxMovie.movieName withExtension:@"mov"];
self.gpuMovieFX = [[GPUImageMovie alloc] initWithURL:urlB];
self.gpuMovieFX.playAtActualSpeed = true;

self.filter = [[GPUImageChromaKeyBlendFilter alloc] init];
[self.filter forceProcessingAtSize:CGSizeMake(640, 640)];

[self.gpuMovieFX addTarget:self.filter];
[self.gpuMovieA addTarget:self.filter];
[self.filter addTarget:self.outputView];

//setup writer
NSString *pathToMovie = [NSHomeDirectory() stringByAppendingPathComponent:
        [NSString stringWithFormat:@"Documents/%@_%@.%@",fxMovie.movieName, self.uuidOfVideo, VIDEO_SAVE_EXTENSION]];
unlink([pathToMovie UTF8String]); // If a file already exists, AVAssetWriter won't let you record new frames, so delete the old movie
self.convertedVideoPath = [NSURL fileURLWithPath:pathToMovie];
self.movieWriter = [[GPUImageMovieWriter alloc] initWithMovieURL:self.convertedVideoPath
                                                            size:CGSizeMake(640.0, 640.0) compressVideoOutput:YES];
[self.filter addTarget:self.movieWriter];    
self.movieWriter.shouldPassthroughAudio = YES;
self.gpuMovieA.audioEncodingTarget = self.movieWriter;
//add this line otherwise it will cause "Had to drop an audio frame" which cozes the saved video lose some sounds
[self.gpuMovieA enableSynchronizedEncodingUsingMovieWriter:self.movieWriter];

[self.movieWriter startRecording];
[self.gpuMovieA startProcessing];
[self.gpuMovieFX startProcessing];

__weak typeof(self) *sself = self;

[self.movieWriter setCompletionBlock:^{
    sself.gpuMovieA.audioEncodingTarget = nil;
    [sself.gpuMovieFX endProcessing];
    [sself.gpuMovieA endProcessing];
    [sself.movieWriter finishRecording];
    sself.recording = NO;
}];

The code is pretty straightforward. There is one problem with GPUImage movie merging, which is it can not merge two audio into final output video, but that’s not the point of this blog.

So when you run the project and compare the video ouput with original video, you would find that the exposure and white balance get little weird, kinda like following picture:

Kinda bizzare. I did search in github and found that there are some issues related to this like #1313 GPUImageMovieWriter changes exposure and white balance and #1306 GPUImageView renders video brighter than AVPlayer.

And author Brad Larson did reply to this question in https://github.com/BradLarson/GPUImage/issues/1306#issuecomment-29392636

That’s most likely due to a slight difference in the way that I convert YUV sources to RGB when loading from movies and video sources. Look at the matrix applied in the kGPUImageYUVVideoRangeConversionForLAFragmentShaderString and the like. I had been using Apple’s standard YUV conversion, then a couple of people changed it, saying that it didn’t match what it should be. Perhaps they were wrong, and the color matrix still needs to be adjusted here.

Then hmmm, I am thinking maybe I should look up for this kGPUImageYUVVideoRangeConversionForLAFragmentShaderString in library and only to find that it is defined in GPUVideoCamera.m class.

 
NSString *const kGPUImageYUVVideoRangeConversionForLAFragmentShaderString = SHADER_STRING
(
 varying highp vec2 textureCoordinate;
 
 uniform sampler2D luminanceTexture;
 uniform sampler2D chrominanceTexture;
 uniform mediump mat3 colorConversionMatrix;

 void main()
 {
     mediump vec3 yuv;
     lowp vec3 rgb;
     
     yuv.x = texture2D(luminanceTexture, textureCoordinate).r;
     yuv.yz = texture2D(chrominanceTexture, textureCoordinate).ra - vec2(0.5, 0.5);
     rgb = colorConversionMatrix * yuv;

     gl_FragColor = vec4(rgb, 1);
 }
);

Nothing special here. I checked its commit history and found out that this commit Re-working GPUImageVideoCamera to use the YUV->RGB transform method s… is very suspicious.

My first action is revert this commit, but failed because in order to make GPUImageMovie support audio play I already made big change to this class. So I just changed the definition of kGPUImageYUVVideoRangeConversionForLAFragmentShaderString the GPUImageVideoCamera.m to original one:

 
NSString *const kGPUImageYUVVideoRangeConversionForLAFragmentShaderString = SHADER_STRING
(
varying highp vec2 textureCoordinate;

uniform sampler2D luminanceTexture;
uniform sampler2D chrominanceTexture;

void main()
{
    mediump vec3 yuv;
    lowp vec3 rgb;

    yuv.x = texture2D(luminanceTexture, textureCoordinate).r;
    yuv.yz = texture2D(chrominanceTexture, textureCoordinate).ra - vec2(0.5, 0.5);

    // BT.601, which is the standard for SDTV is provided as a reference
    /*
     rgb = mat3(      1,       1,       1,
     0, -.39465, 2.03211,
     1.13983, -.58060,       0) * yuv;
     */

    // Using BT.709 which is the standard for HDTV
    rgb = mat3(      1,       1,       1,
            0, -.21482, 2.12798,
            1.28033, -.38059,       0) * yuv;

    gl_FragColor = vec4(rgb, 1);
}
);

Not sure just replacing the shader string is gonna be okay or not, but at least we tried out and the exposure problem solved.

Hopefuly this could shed some light on others when using this library.

iOS7 Tech Talks Shanghai

2013-11-12T15:21:59+00:00

比较幸运的被选中参与了苹果组织的iOS7 Tech Talks上海站的活动，整个活动有两天，分别是iOS7应用开发和游戏开发。我选择报名了App开发讲座，这个讲座会有一整天而且整个安排都非常的紧凑，虽然其中有些东西我在WWDC 2013的视频中有看过，但是总体而言，还是很有收获的，因为它不仅仅只是关于开发，它还有关于UI设计的，市场等等，总的来看就是WWDC 2013的精简版。

简单梳理一下自己觉得有意思或者自己之前不太了解的一些知识点吧。

Interface Design

我印象深刻的是关于按钮上得图标设计。比如按钮一般都有正常和被选中两种状态，每种状态对应不同样式的图标。在iOS6上面，按钮图标正常状态是突出来，而被选中状态的图标就是凹下去的效果(阴影啊渐变啊都不一样)；在iOS7上面可能就不一样了，在正常状态的图标当然肯定木有什么阴影啊渐变等拟物化的效果，一般来说，你最好保持图标的stroke(描边?)的宽度为1个pt，然后尽量保证图标简洁，不要有太多的填充。为什么了？

因为选中状态的图标相对正常状态下得图标有两种变化：

将描边的宽度增加一个pt

填充整个图标的颜色为蓝色或者对应的颜色

其中第二条也是为什么你的图标尽量要简洁，不要有太重的填充，最好是简单是几条线条就能勾勒出来，可以参考Safari，AppStore和WWDC等应用的设计。

总结来说在设计图标时，尽量遵循三个原则：

尽量使用简单的几何图形，比如三角形，圆圈，直线，矩形就能勾勒出图标

保持描边的宽度在1pt

坚持填充部分的比列

对于按钮，iOS7的按钮被大大的简化了，没有拟物化那么多得细节，一般来说对于在导航栏和tab bar而言，按钮需要多余的边框，只需要突出的颜色，比如蓝色就好了，因为这些位置就足以告诉用户这些是按钮而不是标签。

但是对于其他位置的图标，就不能这么简单了，比如你有多个按钮垂直的排列，如果直接使用按钮，用户可能无法知道说这个是可以交互的按钮，可能以为这只是标签而已。你可以使用table view将这些按钮封装起来，这样一来按钮中间有了分隔线，可以帮助用户更容易的区分，同时table view也让按钮的点击区域覆盖了整个屏幕。

对于横向排列多个按钮，一是可以采用table view, 另外一个可以给按钮加边框。

尽量还是参考原声的应用，仔细分析每一个细节，然后思考为什么这么做。

iTunes Connect

这里讲了很多东西，还包括不少iTunes Connect中如何仔细选择app的名字，描述， story-telling的截图，以及如何选取关键字。

描述尽量简洁，最好三四行文字能慨括出app的应用场景。因为一般来说在appstore中浏览时，超过4行就会有一个“更多”或者“展开”的选项，用户很有可能都懒得点开这个，所以你需要简单简洁的描述你的app.
应用截图。最好了应用的截图顺序能够流畅的表达出应用的特点和场景。最好不要在上面加一些多余的什么漂浮的文字。
关键字。尽量不要有重复的字段，或者同时出现单复数，也不要选择app所在的类别。

比如说你有一款音乐应用,下面的这个描述：

Music, Music art, album, albums, tracks , song, iTunes

可以改为:
album, song, track (后面的记不起来了。。。)

现在iTunes Connect中每个应用的每个版本会有100个promo code

现在价格策略中，引入了新的价格策略。比如之前在美国是0.99美元，在中国是6.00人民币的第一价格梯队，有了更多选项，比如在美国仍然还是0.99美元，在中国却是8.00人民币的价格梯队。

Marketing

推广渠道：

让社交应用更多深入的集成到应用中。比如你可以直接从应用中分享内容到社交圈子，或者可以直接可以关注比如官方微博或者微信等等。
使用smart app banner.简要的说就是在web页面的移动版本中只需要添加一行代码，就可以实现当手机访问这个网页时，网页顶部会出现一个banner, iOS会检查这个应用是否已经安装，如果没有，将提示用户下载，如果有，提示用户打开。具体的效果，你可以尝试在safari中访问tmall.com天猫来看一下天猫是如何做得。
在第二条基础之上，保证有多个可以提示安装APP的入口，这样一来用户可以随时方便的安装app. 比如在第二条的网页中的smart banner,永远会待在顶部，当你滚动页面时，可能就看不到了，你需要在页面的其他地方，比如中间和底部加安装应用的入口。
使用内置的商店表单。比如一般来说应用或者游戏都会有一个”more games/apps”，当用户点击这个按钮，会在应用中显示app store,并且展示公司下所有的应用或者游戏。
Trailer预告片。
- 预告片的头六秒钟是最重要的；
- 尽量保持长度在30到60秒；
- 使用专业的视频捕捉工具；
- 尽量选一个更好的音乐轨；
- 保证有明显的比如“download in appstore”下载的点击提示
多种下载入口。比如可以使用下载徽章或者使用二维码，因为在国内二维码扫描下载可能更加方便。

iOS6 Compatibility Coding Strategy

如何保证向后兼容性了？从开发的角度来看有几点：

框架。保证新的框架在build settings里Link Library中将其标记为optional. 如果被标记为required, 应用在启动时发现没有对应的框架时，会拒绝启动，直接崩溃。
新的类。在新类上面直接调用方法并不会有问题，因为系统会直接将其替换为nil.
Capability. 比如motion和mail,你可以通过[MailManager canSendMail]类似的来检测是否被支持。
方法。可以使用respondToSelector来检测是否被支持。

 
SEL notImpSelector = @selector(notImp:);

if([tester respondsToSelector:testSelector])
{		   
    //call it
    [tester notImp:param];
}else{
	//fallback
}

你可以看到在这个过程中你会发现很多这样的if/else，如果这只是一两处没有问题，但是如果出现了很多次，那么这个就会非常痛苦，特别是维护，当你想删除或者修改内部实现细节时，你不得不一个一个搜索出来并修改。

理论上你不应该去考虑或者太多知道这些关于iOS实现差异的细节，你是app开发者，你关注是你的应用逻辑和实现，底层这些细节应该一种更加透明的方式被封装起来。

为了应对此问题，我们可以采用三种策略来封装这个细节：

1. Class Cluster

这个在iOS中被广泛使用， 比如UIButton,它只有一个通用的interface，但是它的实现中却是有几种不同的实现。举个例子，在iOS7中有新的更强大的[NSURLSession](http://www.raywenderlich.com/51127/nsurlsession-tutorial), 但是在iOS6还是只有NSURLConnection. 假设我们有一个DownloadManager类来封装这个细节，对外它有一个DownloadManager.h有开始，暂停，终止等操作。在DownloadManager.m中代码会使这样的：

 

	@interface DownloadManager_NSURLSession : DownloadManager
	// implementation goes here for using NSURLSession
	@end
	
	@interface DownloadManager_NSURLConnection : DownloadManager
	// implementation goes here for using NSURLConnection
	@end
	
	
	@implementation DownloadManager
	
	+(instancetype)sharedInstance {
	    if([NSURLSession class] != nil){
	        return [[DownloadManager_NSURLSession alloc] init];
	    }else{
	        return [[DownloadManager_NSURLConnection alloc] init];
	    }    
	}
	
	@end

日后你不需要支持iOS6时，你只需要该一处地方。

2.Data Delegate 委托

这个模式那更加常见了，iOS本身很多地方都运用了这个模式，以便减少不必要的继承，继承是如此容易被滥用，更多时候你应该考虑责任划分，比如像tableview, 它有数据来源的委托，也有界面更新事件的委托。下面展示如何使用这个模式：

 
	@interface DownloadManager()
	@property (weak) id<Downloader> downloader;
	@end
	
	
	
	@implementation DownloadManager
	
	+(instancetype)sharedInstance {
	   id<Downloader> delegate = nil;
	   if([NSURLSession class] != nil){
	       delegate =  [[NSURLSessionDownloader alloc] init];
	   }else{
	       delegate = [[NSURLConnectionDownloader alloc] init];
	   }
	   return [[self alloc] initWithDownloader:delegate];
	}
	
	@end

3. 使用Category来封装新方法

在iOS中使用一个category来封装这个细节，比如在iOS7中UIView增加了一个方法addMotionEffect，对于此我们可以创建一个UIView+Compatibility的category, 代码如下:

 
	@implementation UIView (Compatibility)

	- (void)c_addMontionEffect:(UIMotionEffect *)effect {
	    if([self respondsToSelector:@selector(addMotionEffect:)]){
	        [self addMotionEffect:effect];
	    }     
	}
	
	@end

到时候当你不需要时，直接rename就好了。

总的来说，你需要思考我应该支持这个功能吗，如果这个需要花费很多时间，你就需要权衡了，因为目前iOS7的占旅游高大71% 而iOS6也才20%?(这里我不记得会议那个百分比了，这个不是很重要啦),关键是你有时候你根本不需要这么大费周章让向后兼容那么完美。

Useful Coding Tips

使用instrument可以做很多的事情，帮助你分析你应用的一些可以提高的地方。比如Time CPU这个分析，你可以发现比如

 
	[myArray enumerateObjectsUsingBlock:^(id object, NSUInteger index, BOOL *stop) {
	    [self doSomethingWith:object];
	}];

这个在多核时依然会出现遍历是线性的。你可以使用NSEnumerationConcurrent来发挥多核的优势：

 
	[myArray enumerateObjectsWithOptions:NSEnumerationConcurrent usingBlock:^(id obj, NSUInteger idx, BOOL *stop) {
	    [self doSomethingWith:object];
	}];

当然前提是你数组遍历跟顺序木有关系。

其它的关于XCode有多优秀，我觉得AppCode完胜它，就不多解释了，特别是重构的支持。

最后说一下，关于本地化的测试。如果你做过这方面的开发测试，你知道这玩意调试起来很麻烦的，每次需要跑到设置中切换语言，然后重新打开应用。现在你可以是多个scheme来简化这个流程.

题外话

在会议的末尾有一个活动的环节，有机会跟Apple的工作人员交流一下，我刚好有机会跟一个负责app review的工程师聊了聊。其实他们的工作也是一部分自动一部分手动，而且一般来说你只要看了那个app review的册子，基本上应该不会被拒掉。顺便聊了下工作经历生活等等，最后我跟说了一下我的一个想法。因为我们知道XCode上传的binary其实包含几部分，一个是exec可执行文件（指令），一部分是xib，一部分是storyboard,最后还有所有的图片。而且刚好后面有session提到说你应该尽量减少图片的大小，特别是在iPad上面，图片会比iPhone上大很多，有些时候有些相片类图片最好使用JPEG格式。我在想，其实应该理论上有可能说在用户从iPhone/iPad上登录AppStore的话，AppStore是可以得到你设备的信息，比如是否支持Retina显示，如果不支持，那AppStore会聪明动态将应用链接到非retina得图片集，然后下载下去；相反，选取高清图片集，准备下载。虽然说理论上用户有可能在mac/windows上的iTunes来买东东，但是大部分情况下都市通过设备，这样一来可以缩减下载文件的大小。

References

objc.io From NSURLConnection to NSURLSession
NSArray enumeration performance examined
NSHipster Launch Arguments & Environment Variables
Stackoverflow Xcode: Run project with specified localization

Integrate Tapjoy Offerwall To Unity

2013-04-24T14:25:09+00:00

Recently I need to integrate Tapjoy offerwall into app for non-Chinese users in Unity. Well after taking a look at the Tapjoy offerwall documentation, I downloaded the sdk and am about to integrate into unity project. But during that, I just ran into tons of problems.

##Walkthrough So basically I have a script CurrencyManager in my project, which is attached to an non-empty and non-destroyable GameObject- in my case its name is CurrencyManager. What basically Currency Manager is doing is that it stores everything about points like spend points,award points and query points. Nothing special. And it has couples of instance variables like totalPoints,spentPoints and awardedPoints to track points locally and synchronize with remote offerwall service provider’s server(here is tapjoy backend). It is interesting as according to Tapjoy documentation:

Tapjoy managed currency enables you to use Tapjoy’s servers to store your user’s currency amount.

What it says it that Tapjoy is gonna save total points,spent points and awarded points in backend. That means you need to maintain synchronization of points between local and remote. As sometimes,network is not available and user’s any operation on points will not be synchronized to remote tapjoy backend immedidately, and when network is available later, those points should be synchronized properly. And Tapjoy is in charge of everthing about points, it has a setting in admin panel that deal with amount of points you want to give to new user(first install) and configure Conversion rate and add test devices.

Here is simplified version of CurrencyManager.cs :

 

public enum OfferWallType
{
	Youmi,Tapjoy
}
public class CurrencyManager : MonoBehaviour
{
	public static OfferWallType offerWallType;	
	private static CurrencyManager _instance;	
	private OfferwallBase _offerwallService;
	  	
	public void init()
	{				 		
		if(CurrencyManager.offerWallType == OfferWallType.Youmi){
			_offerwallService = (Youmi) gameObject.AddComponent(typeof(Youmi));
		}else{
			_offerwallService = (TapJoy) gameObject.AddComponent(typeof(TapJoy));
		}
	}
		
	public void showOffers()
	{
		_offerwallService.ShowOffers();
	}
		
	public void syncPoints()
	{
		bool spentORaward = false;
		if(spentPoints > 0)
		{
			_offerwallService.SpendPoints(spents);			
			spentORaward = true;
		}
		
		if(awardedPoints > 0)
		{
			_offerwallService.AwardPoints(awardedPoints);			
			spentORaward = true;
		}
		
		//if no spent or award, query points
		if(!spentORaward)
		{
			queryPoints();
		}
		
	}
	
	public void pointsAward(int value)
	{
		awardedPoints += value;
		syncPoints();
	}
	
	public void pointsSpend(int value)
	{
		spentPoints += value;
		syncPoints();
	}
	
	public void queryPoints()
	{
		if(Configuration.isDevice())
		{
			_offerwallService.GetPoints();						
		}	
	}
	
 	public void showEarnedPoints()
	{
		if(Configuration.isDevice())
		{
			_offerwallService.ShowEarnedPoints();
		}
	}
	
	private bool _totalPointsChanged;
	private int _totalPoints;
	public int totalPoints
	{
		get{
			if(_totalPointsChanged)
			{
				_totalPointsChanged = false;
				 
				 //You mean need to encrypt the value before save
				if(PlayerPrefs.HasKey(PREFS_TOTAL))
				{
					_totalPoints = int.Parse(Parse(PlayerPrefs.GetString(PREFS_TOTAL));
				}
				else
				{
					_totalPoints = 0;
				}	
			}
			
			return (_totalPoints + awardedPoints - spentPoints);
		}
		
		set{
			_totalPointsChanged = true;
            PlayerPrefs.SetString(PREFS_TOTAL_HEARTS, value.ToString());
		}
	}
	
	//..do similiar setup for awaredPoints and spentPoints setter and getter
	 
}

CurrencyManager is pretty flexible as you can add other offerwall services quite easily to the code(Like here we also implement YouMi). The method worth pointing out is that for totalPoints,awaredPoints and spentPoints, the setter is gonna actually save points locally and getter is loading points from PlayerPrefs. So that we don’t rely on backend to track points. And everytime you award/spend points, it is gonna trigger syncPoints() to sync local points to tapjoy services.

Of course, our CurrencyManager is singleton.

Here is the tapjoy code:

public class TapJoy : OfferwallBase
{	 
	void Start()
	{		
		TapjoyPlugin.SetCallbackHandler("CurrencyManager");
		
		TapjoyPlugin.EnableLogging(true); //for testing
		TapjoyPlugin.RequestTapjoyConnect(Configuration.TAPJOY_APP_ID, Configuration.TAPJOY_SECRET);
		TapjoyPlugin.SetTransitionEffect((int)TapjoyTransition.TJCTransitionExpand);		 		 
	}
	 
	public override void ShowOffers()
	{
		TapjoyPlugin.ShowOffers();
	}

	public override void SpendPoints(int points)
	{
		TapjoyPlugin.SpendTapPoints(points);			
	}
	
	public override void AwardPoints(int points)
	{
		TapjoyPlugin.AwardTapPoints(points);
	}

	
	public override void GetPoints()
	{
		TapjoyPlugin.GetTapPoints();
	}


	public override void ShowEarnedPoints()
	{
		TapjoyPlugin.ShowDefaultEarnedCurrencyAlert();
	}

	#region Tapjoy Callback Methods (These must be implemented in your own c# script file.)
	// CONNECT
	public void TapjoyConnectSuccess(string message)
	{
		Debug.Log("HandleTapjoyConnectSuccess");
	}
	
	public void TapjoyConnectFail(string message)
	{
		Debug.Log("HandleTapjoyConnectFailed");
	}
	
	// VIRTUAL CURRENCY
	public void TapPointsLoaded(string points)
	{
		Debug.Log("HandleGetTapPointsSucceeded: " + points);
		CurrencyManager.getInstance().totalPoints = int.Parse(points);
	}
	Point
	public void TapPointsLoadedError(string message)
	{
		Debug.Log("HandleGetTapPointsFailed" + message);
	}
	
	public void TapPointsSpent(string points)
	{
		Debug.Log("HandleSpendTapPointsSucceeded: " + points);
		CurrencyManager.getInstance().spentPoints = 0;
		CurrencyManager.getInstance().totalPoints = int.Parse(points);
	}

	public void TapPointsSpendError(string message)
	{
		Debug.Log("HandleSpendTapPointsFailed " + message);
	}

	public void TapPointsAwarded(string points)
	{		 
		Debug.Log("HandleAwardTapPointsSucceeded " + points );		 
		CurrencyManager.getInstance().awardedPoints = 0;
		CurrencyManager.getInstance().queryPoints();
	}

	public void TapPointsAwardError(string message)
	{
		Debug.Log("HandleAwardTapPointsFailed");
	}
	 
	public void CurrencyEarned(string points)
	{ 
 		Debug.Log("CurrencyEarned" + points ); 
 		TapjoyPlugin.ShowDefaultEarnedCurrencyAlert();	}

	#endregion
}

Nothing special, we just followed official sample code and the documentation Tapjoy Integrating the Offerwall. And the Tapjoy class here is like composition, just hide complexity of code that acutally deal with tapjoy.

Here is the thing you may wonder why CurrencyEarned is not called. The trick is that after you set TapjoyPlugin.SetCallbackHandler, everytime response from tapjoy backend like queryPoints or awardPoints, the tapjoy is pretty smart to check whether it is necessary to call CurrencyEarned or not.

##Problem Okay, so you have all the code ready and can’t wait to build out xcode project and run it on your device. Oh, before you test it, don’t forget to set Init Balance and add your device to Test Devices, which first one will give you init points and second one will you make your testing way easier.

You launch the app and open offerwall, click Test Offer (Click to receive 10 points) in the first row and press home button and open you app again, you will see an alert : “Congratulations! You’ve just earned 10 Points!”. Perfect! Everything looks good.

Then where is problem I’m gonna talk about?

Here for a typical game, you have a collectable items in gameplay.Like here we have collectable item when user is playing the game, and user can just pick up the item and their points in lower-left corner will be increased by one. Well user definitely will pick up the item, then after 2 or 3 seconds, BANG, you got an alert view saying “Congratulations! You’ve just earned 1 Points!”. And you lose the control of the game, until you press the OK button of alert view. Then maybe your lovely character is already dead by hitting to rock or stepped on by Chuck Norris. So you may be wondering WTF!

##Solution You may think that we can do a check in method CurrencyEarned

public void CurrencyEarned(string points)
{ 
	Debug.Log("CurrencyEarned" + points ); 
	TapjoyPlugin.ShowDefaultEarnedCurrencyAlert();
}

by checking like GameState, normally game will have following states: None, Playing,Pause, Continue, GameOver. Then how about adding following check ?

public void CurrencyEarned(string points)
{ 
	Debug.Log("CurrencyEarned" + points ); 
	if(GameManager.GameState == None || GameManager.GameState == GameOver)
	{
		TapjoyPlugin.ShowDefaultEarnedCurrencyAlert();
	}		
}

Here for our case is even tricky as we also show it not only when game scene not started/loaded also when it is gameover.

But then we have another problem, it is that character right before it is dead, it pick up a point power-up item. Then you will an alert view saying “Congratulations! You’ve just earned 1 Points!”. Again, you need to press OK button to dismiss it. You may think this case it is pretty rare but when your game has millions users. It is not a neglectable issue.

Let’s revisit the logic. So basically we only want CurrencyEarned be triggered when user lanunch the app. And during the gameplay, we dont’ want it get called.

Maybe we can set a flag like enabled on like TapjoyEarnedPointManager, which should be a singleton. Then in AppController.mm go to - (void) applicationDidBecomeActive:, check currently enabled offerwall services first and see if it is Tapjoy enabled, then calling [TapjoyEarnedPointManager sharedInstance].enabled = TRUE.

Then go to Mono and find your UnityInterface.cs wrapper-which contains all your unity-objectivec magic method delcarations. Add following codes:

[DllImport("__Internal")]
public static extern bool Plugin_IsOfferWallCheckPointsTriggered();
public static bool IsOfferWallCheckPointsTriggered()
{
	if(Configuration.isDevice())
	{
		return Plugin_IsOfferWallCheckPointsTriggered();
	}
	return false;
}

[DllImport("__Internal")]
public static extern void Plugin_SetOfferWallCheckPointsTriggeredToFalse();
public static void SetOfferWallCheckPointsTriggeredToFalse()
{
	if(Configuration.isDevice())
	{
		Plugin_SetOfferWallCheckPointsTriggeredToFalse();
	}
}

And in your unity interface navtive code somewhere add implmenetations:

	  
  ...
    bool Plugin_IsOfferWallCheckPointsTriggered()
    {
        return ([TapjoyEarnedPointManager sharedInstance].enabled)
         
    }
    
    void Plugin_SetOfferWallCheckPointsTriggeredToFalse()
    {
    	[TapjoyEarnedPointManager sharedInstance].enabled = NO;
    }
    
	
#ifdef __cplusplus
}
#endif

Finally go to your Tapjoy.cs and locate CurrencyEarned, change it to following:

public void CurrencyEarned(string points)
{ 
	
	if(UnityInterface.IsOfferWallCheckPointsTriggered())
	{
		UnityInterface.SetOfferWallCheckPointsTriggeredToFalse();
		//it may have network delay and you actually now is playing it, don't alert under that case
		if(GameManager.GameState == None || GameManager.GameState == GameOver)
		{
			TapjoyPlugin.ShowDefaultEarnedCurrencyAlert();
		} 	

	}else{
		Debug.Log("Currencty Earned IsOfferWallCheckPointsTriggered is false, not show it");
	}
}

Build and run it. It should slove previous problem perfectly.

Yup!

##References

Tapjoy Offerwall Unity Plugin

Tapjoy Integrating the Offerwall

Tapjoy Managed Currency

Unity: Automate Post Process

2013-04-11T14:25:09+00:00

Everytime after I build a Xcode project from Unity for PigRush, I need to manually link some system frameworks, some third-party frameworks, add my native code and change some build settings. It is really a pain in the ass. After struggling with Unity’s Build Player Pipeline, finally I got process automated. The problems I found during process should cover lots of scenarios you probably will run into and hopefully this will give you some lights :)

##Case Intro So I have custom libraries and native code reside under ~/UnityWorkspace/PigRush-iOS/Assets/ObjC, which includes following file structure:

├── Chartboost
│   ├── CBAnalytics.h
│   ├── CBAnalytics.h.meta
│   ├── Chartboost.h
│   ├── Chartboost.h.meta
│   ├── libChartboost.a
│   └── libChartboost.a.meta
├── Chartboost.meta
├── FlurryAnalytics
│   ├── FlurryAnalytics.h
│   ├── FlurryAnalytics.h.meta
│   ├── libFlurryAnalytics.a
│   └── libFlurryAnalytics.a.meta
├── FlurryAnalytics.meta
├── RevMobAds.framework
│   ├── Headers -> Versions/Current/Headers
│   ├── Resources -> Versions/Current/Resources
│   ├── RevMobAds -> Versions/Current/RevMobAds
│   ├── Versions
├── PigRushNavtiveCode
│   ├── Appirater.h
│   ├── Appirater.h.meta
│   ├── Appirater.m
│   ├── Appirater.m.meta
│   ├── EmailComposer.h
│   ├── EmailComposer.h.meta
│   ├── EmailComposer.m
│   ├── EmailComposer.m.meta
│   ├── …blablabla
│   ├── UnityNativeManager.h
│   ├── UnityNativeManager.h.meta
│   ├── UnityNativeManager.mm
│   └── UnityNativeManager.mm.meta

The file structure above is not all, it is part of it.

And the post build process that I’m handling is like following:

Import the system frameworks
- Accounts.framework (optional)
- GameKit.framework (optional)
- MessageUI.framework
- MobileCoreServices.framework
- StoreKit.famework
- Social.framework (optional)
- libsqlite3.dylib
Drag into the project all files and folders I listed above from ~/UnityWorkspace/PigRush-iOS/Assets/ObjC
- run find . -name ‘.meta’ -type f -delete* to delete .meta files first
Modify AppController.h to add following instance variable declarations for UrbanArship push notification
```
 NSString *deviceToken;
 NSString *deviceAlias;
 NSString *pushActionURL;    
```

Modify AppController.mm to add header imports

 #import "Appirater.h"
 #import "RDGameCenterManager.h"
 #import "Chartboost.h"
 #import <RevMobAds/RevMobAds.h>

Modify AppController.mm to add [[RDGameCenterManager sharedInstance] disconnectLocalPlayer]; to end of applicationWillResignActive method
Modify AppController.mm to add [Appirater appEnteredForeground:YES]; to end of applicationWillEnterForeground method
Modify AppController.mm to add other code snippets
Finally set GCC_ENABLE_OBJC_EXCEPTIONS to YES in BuildSettings

Imagine everytime, you build from Unity and you have to do those steps, it is very paninful process.

##Solution

Note: You can find complete code in my github repo: UnityAutomatePostProcess.

As Unity PostProcessBuildAttribute reference says,

Add this attribute to a method to get a notification just after building the player.

– which means that we can use this meta tag to register with Unity engine to kick off post build process.Also notice:

This is an editor class. To use it you have to place your script in Assets/Editor inside your project folder.

Pretty straightforward.

Given that we’re gonna play around xcode project file, it looks like using python is good option(as it is built-in supported on Mac). So what this callback script is just simply calling our post_process.py. We create a file CustomPostprocessScript.cs under ~/UnityWorkspace/PigRush-iOS/Assets/Editor:

    #if UNITY_IPHONE
using UnityEngine;
using UnityEditor;
using UnityEditor.Callbacks;
using System;
using System.Diagnostics;

public class CustomPostprocessScript : MonoBehaviour
{
	[PostProcessBuild]
	public static void OnPostprocessBuild(BuildTarget target, string pathToBuildProject)
	{        
		UnityEngine.Debug.Log("----Custome Script---Executing post process build phase."); 		
		string objCPath = Application.dataPath + "/ObjC";
		Process myCustomProcess = new Process();		
		myCustomProcess.StartInfo.FileName = "python";
        myCustomProcess.StartInfo.Arguments = string.Format("Assets/Editor/post_process.py \"{0}\" \"{1}\"", pathToBuildProject, objCPath);
        myCustomProcess.StartInfo.UseShellExecute = false;
        myCustomProcess.StartInfo.RedirectStandardOutput = false;
		myCustomProcess.Start(); 
		myCustomProcess.WaitForExit();
		UnityEngine.Debug.Log("----Custome Script--- Finished executing post process build phase.");  		
       
	}
}
    #endif

Here the pathToBuildProject is like ~/UnityWorkspace/XCode/PigRush-XCode and objCPath is the path referring to the folder that our custom libraries and native code reside in(~/UnityWorkspace/PigRush-iOS/Assets/ObjC).

Then we dive into our magic post_process.py script.

import os
from sys import argv
from mod_pbxproj import XcodeProject
import appcontroller
path = argv[1]
fileToAddPath = argv[2]
    #path: /Users/tuo/UnityWorkspace/XCode/PigRush-XCode-Test1    
print('post_process.py xcode build path --> ' + path)
print('post_process.py third party files path --> ' + fileToAddPath)    
    #Before execute this, you better add a check to see whether your change already exist or not, as if user
    select *Append* rather than *Replace* in Unity when build, this will save you time and avoid duplicates. 
    
print('Step 1: add system libraries ')
    #if framework is optional, add `weak=True`
project = XcodeProject.Load(path +'/Unity-iPhone.xcodeproj/project.pbxproj')
project.add_file('System/Library/Frameworks/CoreTelephony.framework', tree='SDKROOT')
project.add_file('System/Library/Frameworks/MobileCoreServices.framework', tree='SDKROOT')
project.add_file('System/Library/Frameworks/StoreKit.framework', tree='SDKROOT')
project.add_file('System/Library/Frameworks/Social.framework', tree='SDKROOT', weak=True)
project.add_file('usr/lib/libsqlite3.dylib', tree='SDKROOT')

print('Step 2: add custom libraries and native code to xcode, exclude any .meta file')
files_in_dir = os.listdir(fileToAddPath)
for f in files_in_dir:    
    if not f.startswith('.'): #exclude .DS_STORE on mac
    print f        
    pathname = os.path.join(fileToAddPath, f)
    fileName, fileExtension = os.path.splitext(pathname)
    if not fileExtension == '.meta': #skip .meta file
        if os.path.isfile(pathname):
            print "is file : " + pathname
            project.add_file(pathname)
        if os.path.isdir(pathname):
            print "is dir : " + pathname
            project.add_folder(pathname, excludes=["^.*\.meta$"])

print('Step 3: modify the AppController')
appcontroller.touch_implementation(path + '/Classes/AppController.mm')
appcontroller.touch_header(path + '/Classes/AppController.h')

print('Step 4: change build setting')
project.add_other_buildsetting('GCC_ENABLE_OBJC_EXCEPTIONS', 'YES')

print('Step 5: save our change to xcode project file')
if project.modified:
    project.backup()
    project.saveFormat3_2()

Code is pretty self-explanatory.

Because that we need to mess around with xcode project file, we better use some existing script to do it. And there is one I found which is pretty good: Mod PBXProj (The script I refere here doesn’t support change build setings and has some problem with escape library search path, you can download good one from my github).

The most complicted code would be in the Step 3, which we modified our AppController. This is place you probably do:

add instance variables delcarations in AppController.h
add code snippet to begin of specfic method in AppController.mm
add code snippet to end of specfic method in AppController.mm
add methods to end of AppController.mm

Because you probably need modify that file constantly, it would be great there is flexible and easy way to do it.

Let’s look at part of appcontroller.py:

 
    #!/usr/bin/python
import sys
import os

def process_app_controller_wrapper(appcontroller_filename, newContent, methodSignatures, valuesToAppend, positionsInMethod, contentToAppend=None):
    appcontroller = open(appcontroller_filename, 'r')
    lines = appcontroller.readlines()
    appcontroller.close()
    found = False    
    foundIndex = -1
    for line in lines:         
        print line
        newContent += line
        for idx, val in enumerate(methodSignatures):
            print idx, val
            if (line.strip() == val):
                print "founded match method: " + val
                foundIndex = idx
                found = True
        if found :
            if positionsInMethod[foundIndex] == 'begin' and line.strip() == '{':
                print "add code to resign body"
                newContent += ("\n\t" + valuesToAppend[foundIndex] + "\n\n")
                found = False
            if 	positionsInMethod[foundIndex] == 'end' and line.strip() == '}':
                newContent = newContent[:-3]
                newContent += ("\n\n\t" + valuesToAppend[foundIndex] + "\n")
                newContent += "}\n"
                foundWillResignActive = False
        if line.strip() == '@end' and (not contentToAppend is None):
                newContent = newContent[:-6]
                newContent += ("\n\n\t" + contentToAppend + "\n")
                newContent += "@end"                            
            
    print "-------done loop close stream and content: " + newContent                    
    appcontroller = open(appcontroller_filename, 'w')    
    appcontroller.write(newContent)
    appcontroller.close()        

def chartboostAndRevMob():
    return '''
    Chartboost *cb = [Chartboost sharedChartboost];
    cb.appId = @"XXXX";
    cb.appSignature = @"XXX";
    [cb startSession];
    [RevMobAds startSessionWithAppID:@"XXXXX"]; 
    '''
def importHeaders():
    return '''
    #import "Appirater.h"
    #import "RDGameCenterManager.h"
    #import "Chartboost.h"
    #import <RevMobAds/RevMobAds.h>
    #import "FlurryAnalytics.h"
'''

def pushActionInstanceDeclaration():
    return '''
	NSString *deviceTokenString;
	NSString *deviceAlias;
	NSString *pushActionURL;    
    '''
def pushActionDealloc():
    return '''
    [deviceTokenString release];
    [deviceAlias release];
    [pushActionURL release];
    '''

def extraCodeToAddInAppControllerMMFile():
    return '''
//blablabla
- (void)connection:(NSURLConnection *)theConnection didFailWithError:(NSError *)error {
    [UIApplication sharedApplication].networkActivityIndicatorVisible = NO;
    UIAlertView *someError = [[UIAlertView alloc] initWithTitle:
                              @"Network error" message: @"Error registering with server"
													   delegate: self
											  cancelButtonTitle: @"Ok"
											  otherButtonTitles: nil];
    [someError show];
    [someError release];
    NSLog(@"ERROR: NSError query result: %@", error);

}

- (void)alertView:(UIAlertView *)alertView clickedButtonAtIndex:(NSInteger)buttonIndex
{
    NSLog(@"alert button index %d", buttonIndex);

    if(buttonIndex == 1)
    {
        //ok action
        NSURL *url = [NSURL URLWithString:pushActionURL];
        [[UIApplication sharedApplication] openURL:url];
    }

}
//blablabla
'''
    
def touch_implementation(appcontroller_filename):
    # appcontroller = open(appcontroller_filename, 'w')
    # print(" process AppController.mm add imports header")
    newContent = importHeaders()
     
    #starting line of method {
    methodSignatures = []
    #value to append near }
    valueToAppend = []
	#position to add insert at the beginning o
    positionsInMethod = []

    methodSignatures.append('- (void)applicationWillEnterForeground:(UIApplication *)application')
    valueToAppend.append('[Appirater appEnteredForeground:YES];')
    positionsInMethod.append("end")

    methodSignatures.append('- (void) applicationDidBecomeActive:(UIApplication*)application')
    valueToAppend.append(chartboostAndRevMob())        
    positionsInMethod.append("end")
    
    methodSignatures.append('- (void) dealloc')
    valueToAppend.append(pushActionDealloc())        
    positionsInMethod.append("begin")

    process_app_controller_wrapper(appcontroller_filename, newContent, methodSignatures, valueToAppend, positionsInMethod, extraCodeToAddInAppControllerMMFile())    

def touch_header(appcontroller_filename):
    # appcontroller = open(appcontroller_filename, 'w')
    # print(" process AppController.mm add imports header")
    newContent = ''
    #starting line of method {
    methodSignatures = []
    #value to append near }
    valueToAppend = []
    positionsInMethod = []

    methodSignatures.append('@interface AppController : NSObject<UIAccelerometerDelegate, UIApplicationDelegate>')
    valueToAppend.append(pushActionInstanceDeclaration())
    positionsInMethod.append("begin")
    process_app_controller_wrapper(appcontroller_filename, newContent, methodSignatures, valueToAppend, positionsInMethod)

A breif explanation is given if you need to change it later.

add code to begin/end of specfic method, you just need to copy the method signature and add to methodSignatures.append(‘some method signature’)
if the code snippet is like one line you just do it like this valueToAppend.append(‘[Appirater appEnteredForeground:YES];’) ; and if code snippet is pretty long, you better put it in method like def pushActionInstanceDeclaration(): and append it like this valueToAppend.append(pushActionInstanceDeclaration())
to mark the position of begin/end like positionsInMethod.append(“begin/end”)
put content to append inside extraCodeToAddInAppControllerMMFile and pass it as fifth parameter of process_app_controller_wrapper

There you go. You can put the code change in your appcontroller.py and it is pretty easy to make changes.

##Fix Library Search Path When I build from unity and run it in xcode, I got bunch of errors. Among them , one says that

ld: library not found for -lChartboost

ld: library not found for -lFlurryAnalytics

But I double checked the chartboost and flurry analytics static lib are indeed imported and showed in Linked Libraries of build settings.After searched on Stackoverflow, I open the build setting of xcode by going to

"Targets"-> "Build Settings" -> "Library Search Paths"

Then you will see following settings:

"$(SRCROOT)"
"$(SRCROOT)/Libraries"
\"$(SRCROOT)/../../PigRush-iOS/Assets/ObjC/Chartboost\"
\"$(SRCROOT)/../../PigRush-iOS/Assets/ObjC/FlurryAnalytics\"

WTF! The paths pointing to our custom libraries are got escaped. Then I drill down the code of mod_pbxproj.py, and found following snippet:

def add_search_paths(self, path, base, key, recursive=True, escape=True):
	modified = False
    #blabla
	if escape :
		if self[base][key].add('\\"%s\\"' % path):#'\\"%s\\"' % path
			modified = True
	else:
		if self[base][key].add(path):#'\\"%s\\"' % path
			modified = True
	return modified

which you probably notice the escape flag by default is set to True. Then I change that flag to False, that error was gone :)

##Done ? Not Yet But I still got lots errors in Xcode and no clue aobut what’s going wrong. Until that I found there is another PostBuildProcess script from Kamcord. Kamcord is unity package we imported for use of record and share mobile gameplay video.

Kamcord also has PostBuildProcess script under Assets/Editor folder, which is like following:

public class KamcordPostprocessScript : MonoBehaviour
{

	// Replaces PostprocessBuildPlayer functionality
	[PostProcessBuild]
	public static void OnPostprocessBuild(BuildTarget target, string pathToBuildProject)
	{
		UnityEngine.Debug.Log ("--- Kamcord --- Executing post process build phase.");		
		Process p = new Process();
        p.StartInfo.FileName = "perl";
        p.StartInfo.Arguments = string.Format("Assets/Editor/KamcordPostprocessbuildPlayer1 \"{0}\"", pathToBuildProject);
        p.StartInfo.UseShellExecute = false;
        p.StartInfo.RedirectStandardOutput = false;
        p.Start();		
        p.WaitForExit();		
		UnityEngine.Debug.Log("--- Kamcord --- Finished executing post process build phase."); 
	}
}

It just executes a perl script, which basically just add Kamcord.framework and related resources to xcode.

Let’s look the Link Library With Libraries, we found that Kamcord.framework is missing! But if we remove our custom script, it is showing.

Now we had two post build process scripts. Pretty bad, as we dont’ know the order it will get executed. And the problem that Kamdcord.framework is missing, maybe is because of orders of execution of scripts.

By taking a look at logs of Unity app, we found actually our CustomPostprocessScript.cs runs before KamcordPostprocessScript.cs. (I will talk about how to check the logs from print in python and UnityEngine.Debug.Log in Unity, which it is handy when debugging).

What I want to do here is make sure our custom script always run after other scripts. Because if we execute our script first, then we have no idea what other script is gonna modify, which possibily screws up everything.

Is there any way specify the order of execution of script ? Yes, from Unity Post Process Mayhem, I found that

[PostProcessBuild(0)] // <- this is where the magic happens
public static void OnPostProcessBuildFirst(BuildTarget target, string path)
{
    Debug.Log("I get Executed First");
}

NB: -10 is a higher priority than 100, the default priority is 1

Cool, then we can go back to our CustomPostprocessScript.cs and modify [PostProcessBuild] to [PostProcessBuild(100)]. Then we make sure our script always run after other scripts.

You can see the benefits of coding our post process in a sepearate file. By doing this way, we make sure when we update other package like Kamcord, no matter when they change in their script, it won’t affect our custom script.

##Tips To look the logs of post build process scripts, like print in python, you can follow steps:

open *Console* from spotlight --> left panel FILES *~/Library/Logs* --> expand to *Unity* --> click *Editor.log*

Then you can see logs of Unity. It is also quite usefull when the Unity is freezing, and you want know whether it is really ‘dead’.

Because the xcode project when built from Unity is like 768 M, and it takes Unity 5~10 minutes to build out, which make the process extremely painful.

I’d like to suggest when testing python script, you probably just move a clean copy of AppController or pbproject file to another folder with git supported. Then you can test your script separately without everytime build from Unity.

Python is quite straightford to pick up, like I just spend several mintues to get familiar with its syntax and be able work on it quite easily. BTW, pay attention to the soft tabs and hard tabs when you get indentation problems.

You can have a complete source code here:UnityAutomatePostProcess.

Enjoy unity!

android notification 多个action时，PendingIntent中extra数据始终一样

2013-04-02T05:18:35+00:00

最近需要在项目中添加Android 4.1 Notification BigPicture 支持，来让这个提醒条更加美观。下面便是代码：

Intent notificationIntent = fillDatabaseContentToIntent(context, cursor1, cursor2, false); Intent shareIntent = fillDatabaseContentToIntent(context, cursor1, cursor2, true);

cursor1.close(); cursor2.close(); if(myDbHelper != null) myDbHelper.close(); Bitmap bitmapForBigPicture = BitmapFactory.decodeResource(context.getResources(), R.drawable.background1);

NotificationCompat.Builder notification = new NotificationCompat.Builder(context) .setSmallIcon(R.drawable.fish_grey) .setContentTitle(contentTitle) .setContentText(contentText) .setStyle(new NotificationCompat.BigPictureStyle().bigPicture(bitmapForBigPicture).setSummaryText(contentText).setBigContentTitle(contentTitle)) .addAction( android.R.drawable.ic_menu_add, “View”, PendingIntent.getActivity(context, (int) System.currentTimeMillis(), notificationIntent, 0, null)) .addAction( android.R.drawable.ic_menu_share, “Share”, PendingIntent.getActivity(context, (int) System.currentTimeMillis() + 10, shareIntent, 0, null));

private Intent fillDatabaseContentToIntent(Context context, Cursor cursor1, Cursor cursor2, boolean openShare) { Intent notificationIntent = new Intent(context, DetailActivity.class); notificationIntent.putExtra(DataBaseHelper.CONTENT_ID_FOR_EXTRAS, cursor1.getLong(cursor1.getColumnIndexOrThrow(DataBaseHelper.CONTENT_ID))); notificationIntent.putExtra(DataBaseHelper.CATEGORY_ID_FOR_EXTRAS, DataBaseHelper.FAKE_CATEGORY_ID_DAILY); notificationIntent.putExtra(DataBaseHelper.CATEGORY_LABEL, cursor2.getString(cursor2.getColumnIndexOrThrow(DataBaseHelper.CATEGORY_LABEL))); notificationIntent.putExtra(DetailActivity.TO_SHARE, openShare); return notificationIntent; }

很简单的逻辑， notification中有两个按钮，一个是查看，一个是分享。然后分享的话，我们在extra中将其To_SHARE设置为真，而如果是查看的话，其值为假。

接下来我们在DetailActivity中，获取TO_SHARE的值来决定是否应该调出分享操作。

Bundle extras = this.getIntent().getExtras(); Boolean isToShareFromNotification = extras.getBoolean(TO_SHARE); if(isToShareFromNotification){ gotoShare(); }

但是我们发现这个 isToShareFromNotification 永远都是false,这个很是奇怪。整来整去都木有想明白，网上搜了搜，果然发现了一个方案:android pending intent notification problem.

The way I solved that problem was by assigning a unique requestID when you get the PendingIntent: PendingIntent.getActivity(context, requestID, showIntent, 0); By doing so you are registering with the system different/unique intent instances.

然后尝试了一下给一个不同的requestId.

NotificationCompat.Builder notification = new NotificationCompat.Builder(context) .setSmallIcon(R.drawable.fish_white) .setContentTitle(contentTitle) .setContentText(contentText) .setStyle(new NotificationCompat.BigPictureStyle().bigPicture(bitmapForBigPicture).setSummaryText(contentText).setBigContentTitle(contentTitle)) .addAction( android.R.drawable.ic_menu_more, “View”, PendingIntent.getActivity(context, (int) System.currentTimeMillis(), notificationIntent, PendingIntent.FLAG_UPDATE_CURRENT)) .addAction( android.R.drawable.ic_menu_share, “Share”, PendingIntent.getActivity(context, (int) System.currentTimeMillis() + 10, shareIntent, PendingIntent.FLAG_UPDATE_CURRENT));

果然就正常工作了。

Android上setAlpha诡异问题

2013-03-14T05:18:35+00:00

当有一次需要在imageView上使用setAlpha来改变当前view的透明度时，发现一个很2的问题.

float opacity = (float) (1.0 - ((float)currentTime/1000.0 - spot.originalStartTime)/FADE_OUT_DURATION);
imageView.setAlpha((int)(255.0*opacity));

在调试中发现，本应该这个imageView的alpha从1(完全不透明)降到0(透明)，然后以后整个imageView应该就看不到了消失掉.

但是却发现imageView是从不透明到透明然后又从透明变回不透明。这个甚是奇怪，因为这段代码在iOS上是木有问题的.

回头想了想，猜测可能是当opacity为负值时，android认为它跟正值一样，所以它显示为不透明了.

实验了一下，确实发现这个问题，opacity为负值时，它的最终表现是跟正值一样的. 吐血了 :(

所以又改了改代码

float opacity = (float) (1.0 - ((float)currentTime/1000.0 - spot.originalStartTime)/FADE_OUT_DURATION);
if(opacity < 0)opacity = 0;
hview.setAlpha((int)(255.0*opacity));	

终于正常了

iOS ViewController内存问题

2013-03-11T00:00:00+00:00

在做一个关于相片的app中，发现app使用起来是越来越慢，甚至就是有时崩溃了最后.我们用instruments打开之后观察内存的使用情况发现，从root vc -> detail A vc 之后内存占用变大了，这个没有任何问题，但是从detail A vc返回到root vc之后发现内存占用率却依然是没有降下来.所以如果你尝试几次root vc -> details A vc -> root vc –> detail B vc –> root vc，这里当然是基于UINavigationController来管理每个controller之间的过渡，就会发现内容占用简直要爆表了.

这个很奇怪，我们当时想也许是从detail A返回之后， detail A的viewDidUnload方法没有被调用，所以它所引用的对象都没有被释放.这个就很奇怪了，我的理解是如果你一旦调用了popViewController:animated（如果你使用Seague，其实质也是调用此方法), 那么对应的detail A viewcontroller的viewDidUnload应该是会调用的，如果不是立刻被调用，也应该是很快。但是却发现一直没有被调用，但是dealloc()方法却是被调用了。于是为了验证我的理解是否准确，我创建了一个新项目，很简单的流程 UINavigationController -> RootVC -> DetailVC，然后调试后发现，每次从DetailVC返回到RootVC，DetailVC的dealloc()几乎是“立刻“”被执行了，但是viewDidUnload却是没有.看了一下api文档,

Called when the controller’s view is released from memory. (Deprecated in iOS 6.0. Views are no longer purged under low-memory conditions and so this method is never called.)

所以这个是其实是在iOS5上只有当出现Receiving memory warning,才会被调用.

反过头来，我觉得那么Detail VC的viewDidUnload是没有被调用，但是dealloc方法应该是被调用了，那么内存占用率应该是降下来才对啊，但是却发现dealloc根本木有被执行.

所以猜测应该是这个对象依然仍然被引用着，导致无法被回收。仔细看了看代码，使用了最笨的调试方式–就是删除所有代码，只保留非常基础的代码之后发现依然没有被调用。接着看了看它的父类，果然发现了问题.

在BaseViewController的viewDidLoad中发现了下面代码：

 
[NSTimer scheduledTimerWithTimeInterval:2.0 target:self selector:@selector(updateSyncStatus:) userInfo:nil repeats:YES];
[self updateSyncStatus:nil];


(void)updateSyncStatus:(id)sender {
     BOOL nowSyncing = ([[SyncManager sharedInstance] uploadInProgress] || [[SyncManager sharedInstance] downloadInProgress]);
     if (nowSyncing!=self.syncing) {
	 	....
     }
 }

你可以发现代码中只有初始化timer，却没有相对的invalidate timer的代码，但是本应该被释放的对象无法被回收，致使内存占用率居高不下.

The dealloc method was not being called if any of the references held by a viewcontroller were still in memory

将初始化timer放到viewWillAppear，invalidate timer的代码放到viewWillDisappear变解决了问题.

Solve OpenSwan(Ipsec) L2tp netlink XFRM_MSG_DELPOLICY error

2013-01-24T00:00:00+00:00

#Setup VPN Recently I bought a vps to setup VPN. The OS is Debian 6. Well I’m using OpenSwan(IPsec) + L2tp and follow the instructions from this website:How To Set Up A L2TP/IPSec VPN In A VPS.

Everthing works fine.

ipsec verify

Prints shows that everything works good.

But I often get this error in /var/log/auth.log and I can’t connect to VPN server:

ERROR: netlink XFRM_MSG_DELPOLICY response for flow eroute_connection delete included errno 2: No such file or directory	

I do search on google and openswan forum. But there isn’t any clear/direct answer to this quesiton.

#XFRM_MSG_DELPOLICY? WTF!!! Until I saw the this post on OpenSwan forum : What’s xl2tpd’s ‘ERROR: netlink XFRM_MSG_DELPOLICY …’ about?

The answer is really brief.

> 2) I have all along been experiencing the behavior in OpenSwan that I think is a documented bug:  When I disconnect my iPhone from the VPN, I need to restart it with /etc/init.d/ipsec restart before I'm able to reconnect.  Is there a known fix to this?  I actually have an idea on how I can set up a password-protected URL to remotely restart it, so in a pinch, I can get that working, but obviously a proper fix would be ideal.
That's a known apple bug, and should be resolved if you use openswan 2.6.33 and xl2tpd 1.2.8

Here is solution: make sure you have correct version of OpenSwan and Xl2tpd.

sudo ipsec --version

Then you get following info:

Linux Openswan U2.6.34/K3.0.0-12-generic (netkey)
See `ipsec --copyright' for copyright information.

Check Xl2tpd version by running:

sudo xl2tpd --version

Then you get:

xl2tpd version:  xl2tpd-1.2.8

So make sure you get OpenSwan 2.6.33 installed.

cd /usr/src
wget http://www.openswan.org/download/openswan-2.6.33.tar.gz
tar zxvf openswan-2.6.33.tar.gz
cd openswan-2.6.33
make programs install

But failed. Because that version is not compatible with current kernel version.

Then I try version 2.6.34 and it got isntalled.

Then you config the /etc/ipsec.conf and ipsec.secrets.

Do remember to reboot to make ipsec work.

After reboot, try /etc/init.d/ipsec restart and ipsec verify and it should work.

Yep!

iOS解决CSV文件的中文编码问题

2013-01-18T00:00:00+00:00

起因

最近一次iOS项目中需要按照CSV格式拼装字符串然后添加到邮件作为附件。但是在这个过程我遇到两个很常见的问题：CSV格式和non-ascii编码及BOM(Byte Order Mark)字节顺序标记。

##CSV格式## 这个比较棘手过程开始于：因为将要发送的CSV字符串中有中文符号（数据统计相关的东西），所以我需要支持unicode编码。
当然最简单的实现方式是:

	
MFMailComposeViewController *emailComposer = [[MFMailComposeViewController alloc] init];	 
NSMutableString *csvString = [NSMutableString string];
[csvString appendString:@"姓名,性别,体重,创建时间\n"];
NSString *timestamp = [dateFormatter stringFromDate:[NSDate date]];
[csvString appendString:[NSString stringWithFormat:@"%@,%@,%@,%@\n", @"黄拓", @"男", @"65 kg", timestamp]];
[csvString appendString:[NSString stringWithFormat:@"%@,%@,%@,%@\n", @"詹姆斯", @"女", @"25 kg", timestamp]];	
[picker addAttachmentData:[csvString dataUsingEncoding:NSUTF8StringEncoding]  mimeType:@"text/csv" fileName:@"statistics.csv"];

我们创建了一个MFMailComposeViewController实例，然后以逗号作为分隔符，’\n’作为换行标记来拼接我们的CSV字符串。在最后，我们设置数据的编码为NSUTF8StringEncoding.没什么特别的。
**然后你迫不及待的从邮箱中下载打开CSV文件，但是发现却是一坨乱码！ **
分隔符貌似没有效果，所有的数据都挤在了一个格子里，而且只有一行。

#####修复

很简单的一个修复办法是将逗号替换为tab(‘\t’)，并且将换行符替换为’\r\n’.如果你不这么做，它就没办法工作。 @_@
下面是一个修复的版本：

		
MFMailComposeViewController *emailComposer = [[MFMailComposeViewController alloc] init];	 
 NSMutableString *csvString = [NSMutableString string];
[csvString appendString:@"姓名\t性别\t体重\t创建时间\r\n"];
NSString *timestamp = [dateFormatter stringFromDate:[NSDate date]];
[csvString appendString:[NSString stringWithFormat:@"%@\t%@\t%@\t%@\r\n", @"黄拓", @"男", @"65 kg", timestamp]];
[csvString appendString:[NSString stringWithFormat:@"%@\t%@\t%@\t%@\r\n", @"詹姆斯", @"女", @"25 kg", timestamp]];	
[picker addAttachmentData:[csvString dataUsingEncoding:NSUTF8StringEncoding]  mimeType:@"text/csv" fileName:@"statistics.csv"];

现在你重新试一下，会发现格式已经正确了，有正确的格子数和行数。
新的问题是：每个格子的中文都是乱码？

##Non-ASCII编码以及BOM##

要解决这些乱码问题，首先我们需要指导为什么我们可以通过向CSV文件的头部插入BOM的方式来消除乱码？长话短说，就是它一个让文件接收方知道发送方使用了何种Unicode编码方式的标记，所以通过它我们可以正确的解析字符流。
通过BOM Wiki，我们指导utf-8的十六位表示是”EF BB BF”. 接着我们可以将其添加到CSV字符串的头部，然后特别需要的是将编码格式从utf-8改为utf-16，否则中文将无法正常显示。

下面是最终代码：

	
NSMutableString *csvString = [NSMutableString string];
[csvString appendString:@"姓名\t性别\t体重\t创建时间\r\n"];
NSString *timestamp = [dateFormatter stringFromDate:[NSDate date]];
[csvString appendString:[NSString stringWithFormat:@"\xEF\xBB\xBF%@\t%@\t%@\t%@\r\n", @"黄拓", @"男", @"65 kg", timestamp]];
[csvString appendString:[NSString stringWithFormat:@"\xEF\xBB\xBF%@\t%@\t%@\t%@\r\n", @"詹姆斯", @"女", @"25 kg", timestamp]];
 NSString *sourceString = [[NSString alloc] initWithFormat:@"\xEF\xBB\xBF%@", csvString];
NSData *encodedData = [sourceString dataUsingEncoding:NSUTF16StringEncoding allowLossyConversion:YES];
[picker addAttachmentData:encodedData mimeType:@"text/csv" fileName:@"statistics.csv"];

你可以很奇怪，我们应该只需要在头部添加BOM标记就好啦，为什么每一行内容都需要添加这个标记？具体原因我无法解释，但是如果不加了，那么显示一行可能没问题，但是如果一旦显示超过四五行的内容，就会出现奇怪的乱码。

Okay 搞定!

Arduino+L298N+蓝牙+nodejs

2012-12-15T00:00:00+00:00

上个周末DIY了一部可以网络在线控制（tuohuang.no-ip.org抱歉的是我现在的服务器是在我自己的联想本子跑的，但是我也不太可能一天到晚开着,我下面会说到用Rasperry Pi替换作为服务器的好处，是的，windows系统太恶心)的蓝牙小车，还是很有意思很有乐趣的一件事。可以先上个图

其实粗略一看其实也很简单，就是有两个直流电机，然后一块控制板加上一个蓝牙传输模块，基本上可以控制小车的前后左右以及速度等等，控制的话，你可以直接通过android手机控制（无需wifi),也可以了网络在线控制。说完了最后是个啥样子，接下来是为什么选择做一个两轮驱动的小车，而不是做一个四轮的或者为什么不是选择做一个自平衡像是赛格威Segway那样的很拽的来做了？答案很简单，对于单片机和电子电路的初学者而言，两轮驱动小车是一个很好的起点，你可以一开把功能做的非常简单，然后在上面增加很多的功能和模块，这个过程也就是一步步深入理解电子原件和原理的过程。当然，另外一个好处，这个应该算是比较便宜的入门选项了，如果你要整个四驱或者自平衡，这个开销可以两轮的高很多。而我，一开始主要是在想在过年的时候带着我六岁的弟弟一起做点有点意思的东东，相对于直接送给他一个玩具然后他玩两下丢了，还不如说让他参与一下这个小车是怎么做的，当然了，这个小车可以那么玩具车高级多了，可以玩出很多花样。与其带着他玩泥巴玩炮竹玩CS，这个也许能激发他的一些兴趣爱好嘛。话说正题，要完成这部车需要了解的一下概念或者知识：

电路和基本的电子元器件
arduino
L298n
蓝牙模块通信
nodejs

首先说一下第一个要点，电子电路这玩意我还是听畏惧的，初中时自己做了一个台灯，结果短路把旁边妈妈的大衣给烧黑了一道，而且的话还是挺危险的，220电压，拜托。另外一个大学时的电子电路这门课，把我听的云里云雾，最后考了60分，把我高兴的不行。一想到这个单片机，就老想到汇编中的寄存器啥玩意的，整一个简单东西能把自己给写晕了，很是恶心。其实第一二个话题还是挺相关的，玩arduino一开始只要有基本的初中物理概念（什么电阻和电压之类）就行，而且的话arduino的电压都是5v，所以很安全，你可以很放心，最多也就是把你的arduino板子给烧了，但不至于威胁你的人身安全。那接下来就是介绍下arduino了，其实也是很简单的，所谓开源的电子原型平台，听起来好像来玩的，具体的定义可以百度下。但是它的好处在于不需要了解太底层概念，你提高了自己的编程语言，非常类似C的风格，很简单。你基本上在arduino的编辑器中写好代码，然后上传到arduino芯片上就可以了，就是一个微控制器。那有了控制器，接下来介绍的是L298n电机驱动板，它可以带一个步进电机或者两个直流电机，这里用的是直流电机，它相对于步进电机来说简单一些，同时也便宜一些。那为什么驱动电机还需要专门的电机驱动扳而不是直接连接到arduino来控制了？这是因为直流电机工作时需要很高的电压（2.5-46v)，而ardunio的电压最高只有5v. 这里需要了解一下L298n的工作原理，才能在接线的时候得心应手，很多时候你拿到了东西或者元件，巴不得立刻动手，其实到头来发现却是摸不着头脑而且浪费时间。所以玩电子元器件时，必须第一步仔细阅读产品参数和说明，不然可能出现板子都给烧了。关于L298n背后的原理，你可以查看极客工坊的文章“关于直流电机 H 桥驱动方案的选择”。关于具体L298n 如何接到Arduino以及电机的说明，这里就不多说了，否则文章都写不完，这里只想说，L298n接线确实是非常多，一开始看起来无从下手，但是其实你如果看了L298n是如何工作的话，你接起线来就是轻车熟路了。（代码会在最后贴上）下面是图：

很好，第一步已经完成，你可以成功的把arduino和L298n连接起来，但是你的能控制它才行啊。我不可能拿个笔记本拉着一个USB线跟在车子后面，所以应该选择一种通信方式了，但是无线是最好了。但是环顾四周，你会发现有蓝牙，WIFI, APC220和XBee等等。对比一下，我发现还是蓝牙模块给力，第一相对其他几种选择蓝牙至少便宜一半，第二虽然蓝牙的通信距离比较短，但是我这个小车也不需要说像是航模一下，在一千米之外还需要控制着，最好的一个地方是它能无缝跟android手机集成通讯，这意味着你只要把从蓝牙模块接到小车上，你手拿android手机就可以控制它了，完全不需要网络或者其他笨拙的中间件（比如还的先从手机发给电脑然后电脑在发给小车）。我选的蓝牙模块基本上分为一个USB蓝牙适配器，一个主蓝牙模块以及一个从蓝牙模块。使用时主蓝牙模块先焊到（需要电烙铁来焊一下）USB蓝牙适配器上，然后再讲蓝牙适配器插到电脑上。向下面的图所示：然后将从蓝牙模块连接到arduino上，因为TX/RX是arduino IDE上传代码到芯片时的所使用的端口，所以你如果在烧写代码时需要先将从蓝牙模块的TX/RX先拔掉：当然了，你可以先单独测试蓝牙通讯，比如我就是先用它测试一下是否能控制PMW的值来控制LED灯的亮度来实现"呼吸“的效果，然后在集成去测试电机。这里你需要了解一下什么是串口通讯（频率等），蓝牙如果一旦连接上就可以看到一个叫做“SLAB_USBtoUART"的串口显示为可用，所有命令都是通过这个串口发送给从蓝牙模块。千万注意要检查一下这个串口是不是已经被使用中，否则你发现你怎么发命令，都木有反应，说不定是你在测试调试的时候打开了连接却忘了关闭连接。蓝牙模块其实还是很简单，但是有些要实际中注意的，蓝牙的芯片驱动CP2012 mac版一般是比较不容易搜索到的，你可以这个网站去找到其windows/mac/linux的驱动（http://www.pololu.com/docs/0J7/3）但是安装去要重启。接下来需要下载一个串口通讯测试工具，windows很简单比如串口通讯助手V2.2，mac底下就是 Serial Tools app等。下面是在mac上测试的画面：接下来便是将arduino,L298n，蓝牙模块以及小车底盘等组装集成起来了。就像这个样子: 接下来的问题是我不能只是通过串口调试工具来操控小车啊，太老土了，我也许想让我远在老家的弟弟通过网页来操作小车啊。这个时候我们可以在我们的电脑跑起来一个nodejs服务，创建一个网页，上面有一些简单的按钮来操控上下左右，同时右边有一个实时的视屏流来观察此时小车的状态。当然这里你可以使用其他技术比如rails,php等等，我这里主要讲讲碰到了一些问题。最重要的，如何要nodejs向arduino发送串口指令？这个必须是可以配置频率以及使用哪个串口以及要发送的指令符。在mac/lunix上，你可以使用内置的stty命令：https://wiki.archlinux.org/index.php/Arduino#stty 比如像是下面的代码：

stty -f /dev/cu.SLAB_USBtoUART cs8 9600 ignbrk -brkint -imaxbel -opost -onlcr -isig -icanon -iexten -echo -echoe -echok -echoctl -echoke noflsh -ixon -crtscts //设置频率等 echo -n "f" > /dev/cu.SLAB_USBtoUART //发送"f"指令到蓝牙串口

然后你把这terminal命令行交给nodejs的exec/child_process来执行就可以发现你的小车有反应了。在windows上比较麻烦一点，你需要去设备管理器去看看SLAB_USBtoUART它使用的串口名字是多少（比如com4). 然后nodejs中exec执行命令行指令“echo f > com4”,这里不需要stty，更重要的是千万不要像mac一样在echo后面加"-n"。 Bingo! 最后一个获取实时视频流。 这个是比较麻烦的一件事，如果你有安卓手机的话就很简单了。这里你去安卓市场下载一款叫做"IPWebcam"的应用，然后打开之后你会发现底下有显示一个ip地址，然后你可以在你的电脑尝试需要这个地址就可以获取实时视频流，借助html的videoplay tag，你可以修改一下你nodejs中网页中它的视频流地址，将其指向为你的安卓手机地址，因为其都在一个内网中，应该是非常快的。最后将你的安卓手机放到可以拍摄到小车合适角度的地方。就可以跳到最后一步。 如何让外网可以访问你内网的nodejs服务器？ 非常简单，打开路由器设置，然后找到转发规则，然后增加一个虚拟服务器，比如8080，同时将ip地址指向你跑nodejs的那台机子，然后重启路由器。但是有木有搞错啊？这个地址是http://101.45.191.182:8080/ 太难记了！有木有简单容易记住的DNS啊。是的，有很多动态DNS服务，你可以注册使用http://www.no-ip.com/的免费服务，然后创建一个转发规则，选中“Port 80 Redirect", 将101.45.191.182和8080分别填好。那以后只要访问 tuohuang.no-ip.org 就可以访问我内网的服务器罗！最后贴上主要的电子器件和淘宝地址，当然了主题是便宜要好用，下面是我的购物清单（包含运费)：

智能小车底盘/寻迹小车/机器人/带码盘/测速/强磁电机/ZK-2WD(http://item.taobao.com/item.htm?id=17614027453) 51块
L298N电机驱动板/步进电机驱动模块/机器人/智能小车(http://trade.taobao.com/trade/detail/tradeSnap.htm?spm=a1z09.2.9.80.lNGirU&tradeID=178162818334442) 31块
蓝牙数传套餐：蓝牙主机+蓝牙从机+蓝牙USB转串口(http://item.taobao.com/item.htm?id=18220892827) 75块
40P双头打杜邦端子杜邦线长21CM 进口彩排线带壳子(http://trade.taobao.com/trade/itemlist/list_bought_items.htm?t=20110530&ad_id=&am_id=&cm_id=&pm_id=) 10.5块
电烙铁+烙铁架+高亮焊锡+焊锡膏6合1 （http://trade.taobao.com/trade/detail/tradeSnap.htm?spm=a1z09.2.9.38.lNGirU&tradeID=262580287254442） 31块
DFRobot Arduino入门套件 (http://www.dfrobot.com.cn/index.php?route=product/product&path=119&product_id=531) 130块

总共下来差不多不到350块，但是除了其中的小车底盘，其他器件都是可以重用的。

代码详细可以参考我的github:(https://github.com/tuo/Arduino2WDCars) 里面有arduino代码以及nodejs服务器的代码。

Android listview remember selection and set default selection

2012-10-28T00:00:00+00:00

From an iOS developer’s perspective, I find that it is extremely hard to apply features like “set default selection when starting” and “remember selection status after user clicked row” to ListView.

So let’s start with “remember selection” first.The problem is that even if you know that you can use selector xml to define highlight/pressed/focus style.But that style will not be kept after user clicked that row. For instance, I have a highlighting selector xml (list_selector.xml under res/drawable folder) like this (but you may have other fields need to highlight like text color of textview in row):

<?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">
    <item android:drawable="@drawable/list_selector_pressed" 
		android:state_pressed="true" />
    <item android:drawable="@drawable/list_selector_pressed" 
		android:state_selected="true" />
</selector>

and list_selector_pressed.xml which defined the highlighting style–set the background color to a gray color :

<?xml version="1.0" encoding="utf-8"?>
<layer-list xmlns:android="http://schemas.android.com/apk/res/android">
    <item>
        <shape xmlns:android="http://schemas.android.com/apk/res/android">
            <solid android:color="@color/dark_gray" />
        </shape>
    </item>
</layer-list>

So as @David Hedlund suggested:

Rather, assign an OnItemClickListener, and have it store away the id of the selected item into some variable.

you need to create a instance variable on top of your class:

        private View currentSelectedView;

then go to

@Override
public void onListItemClick(ListView l, View v, int position, long id) {
	if (currentSelectedView != null && currentSelectedView != v) {
        unhighlightCurrentRow(currentSelectedView);
    }

	currentSelectedView = v;
    highlightCurrentRow(currentSelectedView);
    //other codes 
    }

Pretty simple: we check if currentSelectedView is null or current clicked view or not. we first to unhighlight any style by calling method unhighlightCurrentRow(currentSelectedView)—you may wonder why we pass instant variable currentSelectedView as parameter, I will explain it later. Then we assign view to currentSelectedView and highlight current row; so that the style will persist after user’s clicking is done.

private void unhighlightCurrentRow(View rowView) {
    rowView.setBackgroundColor(Color.TRANSPARENT);
    TextView textView = (TextView) rowView.findViewById(R.id.menuTitle);
    textView.setTextColor(getResources().getColor(R.color.white));
}

private void highlightCurrentRow(View rowView) {
    rowView.setBackgroundColor(getResources().getColor(
            R.color.dark_gray));
    TextView textView = (TextView) rowView.findViewById(R.id.menuTitle);
    textView.setTextColor(getResources().getColor(R.color.yellow));

}

Aha, that’s it. That is how we implement “remember selection” for list view. As you see, we have to duplicate the codes for styling both in xml and java code–pretty stupid :(

Next about “set default selection”. You may think that you can do this

	listView.setAdapter(adatper)
	listView.setSelection(0);
	currentSelectedView = listView.getChildAt(0);
	highlightCurrentRow(currentSelectedView);

in onCreate() in activity or onActivityCreated() in fragment.
But if you run it , you will get NullPointer exception and why ? because at this time, the listview is not rendered yet and Android doesn’t like iOS which have viewWillAppear. SO you have to create an instant variable to remember whether it is first time to render listview cell and in onListItemClick to unset that variable:

So under currentSelectedView declaration:

    private Boolean firstTimeStartup = true;

then add methods : suppose we want to highlight the first row in list view:

public class HomeAdapter extends ArrayAdapter<String> {
	int layoutResourceId;

	public HomeAdapter(Context context, int textViewResourceId,
			ArrayList<String> objects) {
		super(context, textViewResourceId, objects);
		layoutResourceId = textViewResourceId;
	}

	@Override
	public View getView(int position, View convertView, ViewGroup parent) {
		if (convertView == null) {
			convertView = LayoutInflater.from(getContext()).inflate(
					layoutResourceId, null);
		}

		if (firstTimeStartup && postion == 0) {
            highlightCurrentRow(convertView);
        } else {
            unhighlightCurrentRow(convertView);
        }

		TextView title = (TextView) convertView
				.findViewById(R.id.menuTitle);
		title.setText(getItem(position));
		return convertView;
	}
}

Pretty simple. But you need to make some changes in onListItemClick method:

@Override
public void onListItemClick(ListView l, View v, int position, long id) {

	if (firstTimeStartup) {// first time  highlight first row
		currentSelectedView = l.getChildAt(0);
	}
	firstTimeStartup = false; 
	if (currentSelectedView != null && currentSelectedView != v) {
        unhighlightCurrentRow(currentSelectedView);
    }

	currentSelectedView = v;
    highlightCurrentRow(currentSelectedView);

     //other codes
}

There you go! Enjoy Android :)

转： e络盟-2012树莓派-raspberry-pi发烧友见面会

2012-10-09T00:00:00+00:00

http://www.element14.com/community/blogs/cn_element14/2012/09/28/e%E7%BB%9C%E7%9B%9F-2012%E6%A0%91%E8%8E%93%E6%B4%BE-raspberry-pi%E5%8F%91%E7%83%A7%E5%8F%8B%E8%A7%81%E9%9D%A2%E4%BC%9A 玩过树莓派 Raspberry Pi ？是树莓派 (Raspberry Pi) 应用高手？如何更好的使用Raspberry Pi？10月17日，我们在上海一起追Raspberry Pi 顶尖ARM解决方案供应商英蓓特Embest专家将受邀出席与我们一起解析Raspberry Pi 也欢迎您展示您的Raspberry Pi应用，我们将为您提前准备投影仪等设备。 e络盟 2012树莓派 (Raspberry Pi)发烧友见面会时间：2012年10月17日（周三）晚18:30 – 19:30 地点：上海浦东碧波路635号传奇广3楼 IC咖啡（馆）如您确认参加本次活动，请email报名申请，确认后我们将给您发邀请函并为您安排简餐及饮用水一份，活动免费。咨询电话：（021）6196 1388 - 1122

毕业两年记（上）

2012-07-14T00:00:00+00:00

前几天看到西门大牛，在人人网上感叹，一晃毕业已经两年了。是的，想一想确实毕业两年了，从2010年七月十二号参加工作到现在也刚好两年了。两年中，收获了很多，成长了很多，更多的是明白了自己更想要什么东西，中间有很多挑战，有很多心得，虽然不是每一个挑战完成的很漂亮，但是挑战的过程我都坚持过来了。最重要的是感谢这两年中帮助和批评过的我人们。 2010/06/28 - 2010/07/12 刚刚毕业没几天吧，印象中，大学对“工作”这个词既有期待，也有未知的惶恐，惶恐以自己不羁张扬的个性能否和公司很好的取得同步，惶恐自己能否能即时调整自己，更加惶恐的是是否这个实际“工作”使我喜欢的。其实我想想当时自己的“惶恐”还是有几分自信的，想起大学，想起袒胸露背喝着果啤一边看港剧的波哥，想起每晚都会准时跟老婆通电话的王维，想起踢实况的乔龙想起搞程序的“土狗”大牛西门云，想起源大牛，想起晚上黑不溜秋的一起在篮球场打球的骚祥，这是生活和学习。身为班干部，确实带头跟老师唱反调，帮童鞋逃课，因为我内心很反感老师的上课方式，我很是怀疑他们教我们的思维方式，我们现实中有用到过吗？所教的知识老套的不行，跟工作中完全脱节，上课方式呆板，知识滞后，在这个日新月异的软件行业而言，老师们的脑壳子明显是停留在是几年前吧。守着陈芝麻烂谷子，却不去尝试了解现实中的前沿科技和变化。我无法忍受是缺乏激情。上次去太原，跟胡总见了一面，聊到说大一时法律政治课，叫大家上台发言，我很屌的上去说了一句“生活就是需要激情啊”。我都不记得了。大二时我开始了逃课，课外自己努力逼迫自己看技术原版书，了解最新的技术，我知道可能一下子很痛苦，但是将来是会有帮助的。而且我现在回头看，确实很有帮助。后来大学几年我的学习成绩直线下滑，但我却很高兴。找工作时，我有明确的目标：敏捷，英语，结果就顺利的进入了Thoughtworks. 2010/07/12 - 2010/09/30 : 西安这天和其他Thoughtworks应届生一起开始了“工作”的第一天。我依稀记得带着我现在还在手头的一个联想笔记本，就是呆呆的坐在一个角落，没有过多的寒暄。后来坐了一会，和几个新同事开始入职介绍。我属于一开始很闷骚的人，不熟的人很是拘谨，但是一聊开了，就是露出本色了。一开始是在凯哥所带的项目上面，一个.NET平台。其实我在大学做的都是一些demo的小项目，说白话就是个人玩玩，所以其实对真正现实世界中的软件项目是什么样的，并没有很深入的了解。对于敏捷，大学时我是看过不少这方面的书，甚至给班上同学做过两次关于重构(Refactoring)，测试驱动开发（TDD)和面向对象设计原则（Robert C Martin那本3P书）的讲座。所以说理论大于实践，可以说一开始进去的第一个月，很多新的挑战，新的知识，我需要迅速学习消化并且应用到项目上。所以那一月，几乎每个晚上回家之后我仍会坚持课外学习，我有个习惯，先查阅很多东西，然后记下要点，等到周末然后系统的总结写成博客，这当然是也是非常感谢凯哥和熊节(gigix)的帮忙。这一点到现在我觉得非常感谢，等到八月份也就是入职后一个月，开始比较能顺心顺手的工作了，每周会和同事去电子正街人人乐旁边的学校打球。是的，在这期间，我大部分时间中午晚上都会做饭，而且还学会了掂锅，呵呵。在第一项目上我一直待到了九月底。 2010/10/01 - 2010/11/16 : 印度，班加罗尔这一次是公司组织的应届生去印度班加罗尔的分公司进行一个关于敏捷课程的培训。学习之余体验了一把异国风情，体验了一把不同文化。待续

记江苏之行（二）

2012-04-23T00:00:00+00:00

从五点半一直捡到七点半，这时天色已经完完全全黑了下来，我还有一段路没有捡完，但是实在是没法继续下来，我提着桶开始往回走。然后在农家乐住了一晚，第二天又赶去了兴化市区，做大巴去了泰州。去泰州，是因为我想在泰州逛逛之后直接做下午三点半从泰州到上海的大巴回去是的，我逛了逛泰州老街，太水了，天气热的很，感觉没啥好看，商业化太浓了。于是我去了坡子街，买了双鞋子，因为之前的鞋子因为前一天在菜花里走来走去走得已经不成样了，确实也穿了好久了，平时也没什么时间去买什么鞋子，上海人太多了，压马路这种事太嘈杂了。结果逛来逛去买了一双2B的红黑配的鞋子，不过看起来还蛮有质感，红黑配这种色调我一直很喜欢，依稀记得大二时的实习我做了一个以红黑配色为主题的网站，结果被人说成了黄网，人家说黄网都是这种色调搭配的，其实我内心还是很纯洁的，根本没想那去。玩的是蛮high。但是。。。。。。我在去泰州的路上上微博，突然看到新闻说上午十一点有一辆从扬州到上海的旅游团的大巴在沿江高速上因为爆胎穿过了防护墙跟对面的大货车迎面相撞，结果造成了几人死亡和几人重伤的事故。看到微博上得配图，非常惨，我顿时感觉后背一股凉气。刚才几分钟前还是活生生的高高兴兴的人们，下一秒却是阴阳相隔，而这个事故离我在的地方（我在泰州）也不是很远，那么近距离虽然不是几米而是上百公里，依然觉得非常震撼。我也是旅游出来周末放松，或许他们也应该是的，而且应该是看完牡丹花蛮开心的返回上海，我也是刚刚看完油菜花也是蛮开心的正在返回途中，以前新闻上看到也不会特别留意没太多触动，但是这件事情跟我的情况是那么相似，我几乎能感受和他们出事前一样的感觉，生命何其珍贵，每个人的生命都只有一次，这些深深触动了我。想起之前我还对周一可能有可能要处理棘手的技术问题而有点烦，但是现在我觉得这都不是问题。我决定改道，从泰州去了无锡。在路上，我还在想着拿起事故，也想起了乔布斯之前在斯坦福的毕业典礼上得演讲，生命留给我们的时间是那么宝贵，如果我们不听从内心的选择，做自己想做的事情，或许当你突然离世，你也许真的什么也没有给这个世界留下。

”活着就是为了改变世界“。

我还很年轻，也许我的经历还是很嫩的，但是我觉得是的，人是应该在适当时候停下来，仔细想想到底自己喜欢这份工作吗，还有当初的那份激情吗？如果当你觉得回答不了自己，也许就是应该做出选择的时候了。我之前做的是互联网领域，技术做的还不错，但是到了一定时间，我觉得我已经找不到当初进公司的那份激情了，激情对于我有多重要？我想起了，当初刚进大一时第一次上思想政治课，老师要同学上去讲人生观，我当时上去直接说”生活就是要有激情啊“，看起来当时还满拽的。我认为这个激情就是做自己喜欢做的事情。

You've got to find what you love. --steve jobs

更重要的是：只要你想好了，年轻没有什么是不可以损失的，不要思前顾后太久太多。我一哥们最近打算换工作，咨询我的意见，我说你如果自己现在不喜欢自己的工作，只是把他当成养活自己的工作，我觉得你看准了自己想干什么，直接去就好了，不管它大或小。你现在想最坏的情形是什么，即使三个月后你失业了，但是这个三个月的经历，有了激情的浇灌，再重新找工作或许还有更好更合适你的工作了。没什么可损失的嘛，看好了，就直接去试。话说回来，去了一趟无锡，去了一趟了惠山老街。很安静，无锡给我的感觉是就是绿色，旁边好多树啊，感觉非常不错，:-( ，可伶的DJN童鞋的合肥啊.... 做自己想做的事情，顾及别人的感受（不自私），不去理会别人的眼光和评论，坚持自己内心的选择。

记江苏之行（一）

2012-04-23T00:00:00+00:00

上个周末去了江苏一趟，这一趟确实是非常值得的。距离这一次出行，我有大概接近一个月没有出去转转了，一直忙于很多事情，也确实让我感觉有点疲惫，于是趁着这个五一来之前，赶到非常热而且人多的五一之前，我决定出去一趟（后来证明我的决策是非常英明的，兴化撑船的大妈说上周几万人，连车都开不进来，这周明显人少了好多，到了下一周或者五一了，最佳看花的时间也就过了）。本来我的想法是去嘉兴西塘的，然后准备在那住一晚上，但是我定的太晚了，所以没有任何的客栈有床铺了。刚刚好，也是看到了林逸欣的微博，想起来她原来当过旅游的外景主持，又看到了她在江苏的片段里面刚刚好讲到兴化剁田的万亩油菜花，而且油菜花在四月份看是最合适的，到了五月份花就开始了凋落了，而且五一人超多超热。所以我下定决心去兴化。周五晚上坐动车去了镇江，等我下车时已经是晚上快10点了，而且下面下着瓢泼大雨，我打车来到了镇江的西津渡历史古街。整个古街都在淅淅沥沥的雨中显得非常朦胧，而且整个街上空无一人，这可是梦寐以求的画面，没有喧嚣，我一个人撑着伞瞎逛着，其实也蛮慎得慌的。刚好看到一家酒吧，于是我赶紧进去，要了瓶啤酒，喝着啤酒，听着风雨声，感觉卸下了所有的包袱和疲惫。第二天一大早，根据我的计划，是一大早又逛了逛西津渡古街，因为晚上太黑了，没怎么看清楚。早上很安静，雨也停了，我走在街上，慢慢悠悠的很快就逛完了，因为街实在是很短。看了一个什么镇江英使馆的建筑，还有点意思，刚好旁边有一个镇江博物馆，顺手过去看了看。但是进去之后发现太令我失望了，整个博物馆喧嚣声一片，很多孩子在跑来跑去大声喧哗，家长则在旁边自己聊自己的，感觉已经把这当成了小学了。我跟保安说了一下，让他找老师说一下让他们安静一些，基本的公德也该是有教的吧，结果根本连老师也找不着。我看了看，实在是太吵了，没法看，也就走了去汽车站赶从镇江到兴化的班车。镇江给我的感觉是非常安静节奏缓慢的一座城市，不错。做从镇江到兴化的班车有了将近三个小时，终于在下午两点到了兴化市区，一路上给我的感觉就是水多树多花多，江苏真是好地方啊。到了兴化市区，急急忙忙赶着从兴化去缸顾乡的班车，因为兴化剁田其实不在市区，而且在离市区比较远的村子里。到了下午四点，终于了到了缸顾乡的千岛菜田。一下车，放眼望去一片黄片的情景，真是蔚为壮观。其实油菜花，还是很常见的，不管南北哪都有，我家里的镇子上也有，只不过是零零散散的种着，很少有大片大片连起来的。小时候，比较喜欢去油菜花田里捉蜜蜂，两只手各拿一张纸，然后看到油菜花上的蜜蜂，悄悄的靠上去，突然一下子把两个手掌合起来，就抓住了，屡试不爽。油菜花吃起来也有讲究，一般会把花去掉，但是油菜花茎秆顶部部分，吃起来比较苦，这个我是深有感受的。我跟我妈说我去看油菜花了，我妈说不就油菜花嘛，这有啥好看的，家里一大片不？其实我或因读书或因工作的缘故，几乎没有在四月份是在家的，油菜花的近距离接触已经好久没有过了，也是一种回味童年吧。下了车，我就迫不及待的检了票，然后去看菜花落。站上高台，放眼望去，整个充斥在眼中的全部都是黄油油的菜花，这是万亩连延不绝的菜花啊，感觉真是被震撼到了，非常漂亮的景色。整个黄色的世界，被青色的河水分成了一个个小方块，看起来好像“井/田”字。当然坐上小船，欣赏着两岸的菜花必然是必不可少的项目啦。黄油油的菜花中有一条小道供游人行走参观，顺着这条小道，仔细品味着两旁近在咫尺的菜花的香味，感觉非常奇妙。其实这条小道不算很长，花了没多长时间，我就已经逛完了。但是走在菜花包围的小道上，虽然两旁景色迷人，但是还是有很多很刹风景的地方：垃圾。道旁边两旁，到处到游人随意丢弃的冰淇淋盒子塑料袋子卫生纸矿泉水瓶等等，非常非常的刺眼和让人不舒服。当第一次走时，我还悄悄的捡起了一两个瓶子，然后给扔到了垃圾桶了。一边走，一边内心在想：“或许应该做些什么，也许我可以就这么走了，但是这一片风景在以后也会很快忘记；因为你没有真正融入它，你没有参与感，你得到的感觉或许不是最纯真的最原始的‘。一个奇怪的想法一直在心里盘旋着，我一直内心在激烈斗争着。当我走回第一次路线的起点时，我看到旁边的警务台旁边有一个红色的塑料桶，我想到这可以是我可以用到的。这个时候天色开始变黑了，而且人慢慢变少了，我内心的斗争也稍微平复了。内心的斗争源自于我是一个人，我还是很怕周遭人的看法，我表示我虽然我有时候比较特立独行，但是无可否认我还是比较留意别人的看法的，任何人都是这样的。这个时候人少了一点点，我想到了如果别人问起我，我可以假意说我是挂名在景区做义工的；如果别人拿很怪异的眼神看我，我就故作镇定一定的装的很像一个真的。然后我迈出了第一步，我把卫衣的帽子翻了下来，盖在头上，在两旁弯着腰一路上捡着垃圾，真是什么垃圾都有，而且有些扔的非常隐蔽，我都不知道他们如何扔进去的，那得是练到了一定功力的“修养”极高才行，轻轻一抛，确是恰当好处，别人还真是难以看到。捡了一阵子，碰到游人，有得会说你是志愿者吗，我木有看他/她，我虚了，我说“是”；其实我在想“沃勒个去，我还能咋回答，我说我是冒牌的志愿者？”。一开始有些拘谨，看到别人怪异的眼神总觉得别人眼神好像在说“装B男”，是的，换做我，第一感觉必然是我也会这么看自己。但是有时候在想，我们看到一些人有些独特的地方或者干了一些别人没有勇气干出来的事情时，我们习惯打击他/她这种“独特”，因为我们无法承认我们自己是没有“特色”的，所以我们想把他拉下来到我们这个等级来，好维持我们仅有的自尊和自信。而缺少特点的人，缺少勇气干自己想干事情的人，缺少勇气听从自己内心深处呼呼的人，在现在的社会中太多了。而一个理想的社会是鼓励每个人的特殊，应为这是他在这个社会无法被替代的唯一性，他得价值。慢慢的，我在垃圾中忙碌开来，已经完全投入了，看到对面来的游人之类的，我都根本没有任何当初的拘谨和面子感，如果我带了一个红色的有义工标志的什么袖子，会让我感觉很体面点，更多是因为我觉得是冒牌的。哪有穿成这样，背着一个书包，当义工检垃圾的，我觉得我给他们丢脸了。捡了好一阵子，大概四分之一吧，累的不行，腰的提不起来，想我打篮球也算是腰力不错的了，各种滞空，哈哈，但是那个时候也是感觉不行了，于是我停下来，握住一把油菜花，把脸贴过去，狠狠吸了一把它的香味，感觉立马又有活力的感觉，而且这个时候路上几乎没有游人，天色也黑了不少，但是还是有点残晖，这个田野感觉顿时安静了下来，天地万物，万籁俱静，整个黄油油的一片包围着着我，放佛要将我吞噬，我顿时感觉这才是我真的追寻的那种似曾相识的感觉，那种安静和那种感觉真像我当年初中时下大雨我把雨衣给住的比我远多的童鞋后我一个人骑着单车在瓢泼大雨中享受着淋透逛哮的感觉，一种透彻心扉美好和原始的纯真。相比起第二天能有看花时可能能有更好感觉的游人们和我的初中童鞋，我觉得我收获的快乐和幸福比他们都要多得多。我开始明白消防战士的感觉，当你看到你抢救出来的财产或者人们，因为你看到因为你的存在，别人的生命能继续或者生活状态能有改变，你的内心会告诉你做了正确的事情，任何你经历的痛苦都会被巨大的幸福感吞噬，你内心的幸福感已经远远大于任何物质诉求，虽然这份工作很危险工资也不高。这却是一份意外的与心灵的对话。

合肥之行

2012-04-14T00:00:00+00:00

距离上次去合肥也有两三周了，今天突然想起来可以记一笔流水账。想去合肥，是因为我之前读过梁启超的《李鸿章传》，里面详细描述他的生平主要是加入湘军曾国藩手下开始一步一步做到开创淮军，打败太平天国，直至真正入阁参与了洋务运动以及各种谈判。更重要的是张辽是我在三国中非常喜欢的人物，当年在逍遥津八百人打败孙权十万人，至今想想都是热血沸腾。在此之前，我对李鸿章的印象依然提留在初高中的课本上面，总觉得他是一个“卖国贼”非常懦弱的人。于是，做为一个2B青年，有时候为了美化自己也时常文艺一下。去过杭州，主要目的除了见同学之外，是想去拜访于谦，也顺便看了看龚自珍的故居。所以，去某个城市逛逛之前，我都会特别留意一些有人文特色的地方。

合肥之行 March 2012

非常不错的是，一路上都有美女相伴，因为我大学童鞋的高中女童鞋刚刚好在安徽大学读研，于是我就麻烦她当我的导游了。安徽大学：

第一站：三河古镇

周六上午去了这个三河古镇，其实谈不上古镇，因为咋们都清楚哪有古镇啊，只不过近几年翻新而已；而我非不太欢喜大城市的喧嚣，我比较喜欢安静的地方，至少人少点的地方。三河镇离合肥有点远吧，坐车也做了将近一个多小时才到。看起来还蛮安静的，整个镇子其实也就是一条河穿过，非常短。尝试了一下这边的米酒，还不错。然后就是来到了一个清代风格的小楼：手贱的不断想拍出七彩：然后就是玩啦，围着小镇转，非常不错的阳光，春天。 DJN童鞋你太高了。。。。。。。

第二站:李府

下午匆匆从三河镇坐车回来，然后去了李鸿章府，是的，我想不通的是李府居然在合肥最繁华的步行街，意味这我也加入了压马路的大军，然后破解一切障碍，终于在一个不起眼的而且在一群现代化建筑中显得非常炸眼的古门牌地方找到了。话说其实也没啥好看的，只不过来缅怀一下。

逍遥公园

这本是我这次来合肥最大的一个目的之一，我想见见张辽当年大战的古战场,结果一进去公园，一开始就是什么各种儿童乐园，各种嘈杂，我这是吐槽了。然后好不容易来到张辽的衣冠冢（谁知道了），结果牌子也没看到，就是光秃秃的一个坟头，旁边写着“请不用登上坟头各种践踏”，我当时就崩溃了。好吧，来公园真是失败。

庐州太太

DJN童鞋说她作为东道主，然后有童鞋来合肥，吃的了，一定得去庐州太太，比较有合肥的特色。悲情的是，当我们如同《功夫足球》中周星驰在风沙漫天的足球场匍匐着像战士一样似的穿过合肥一条灰尘漫天的路，来到了庐州太太面前。结果人爆满，我俩来了一个号，居然是90几，然后我两在旁边坐着玩手机，大概半个小时过去，依旧还是四十几号，吐槽了。然后我们就去旁边的一家XXX餐馆，吃了个牛蛙（这牛蛙真可伶，类似韩寒写的差不多，我们都是牛蛙，上面在闹的都是特权阶级的事情）。然后吃完差不多就是送DJN童鞋回安大了，怎么说这点绅士风度还是应该有的，特别是晚上。

包公祠

如果这次中最坑爹的就是这个包公祠，简直就是坑爹爹爹。。。。。。。我不想说了。，。。。。真是看不到多少真的古迹了，完完全全就是一个仿造。吐槽！！！！伤不起，文艺青年！

庐州太太

是的，从包公祠回来，又去了昨晚的泸州太太，终于人品爆发，等上了。这鱼真是超赞。

晚秋

吃晚饭，去电影院看了晚秋，太坑爹了,有木有。我这种文艺青年表示看不懂，整片有点抑郁。

回程

整个过程，非常感谢DJN的陪同和导游，非常有亲和力，也是有点文艺范。。。。。赞，你！

problem with adding custom font in Xcode 4.3.2

2012-04-13T00:00:00+00:00

Well, I guess John Muchow's blog Load and Access Custom Fonts" pretty much said everything about adding custom font before xcode 4.3.2. Before Xcode 4.3.2, when you add custom font, you need to make sure name of font you use in following code snippet:

[label2 setFont: [UIFont fontWithName: @"BlackBoard" size:18]];

Where is my font ?

the "BlarkBoard" is the real name when you install it on mac. So the file name you just add to Resource and plist file , like "BlackBoard.ttf" is always the same the name of font that installed on mac.So you need to open that font file in FontBook, and write the name showing up down, like "Black BoardSb". Then when you want to refer to font in the code:

[label2 setFont: [UIFont fontWithName: @"Black BoardSb" size:18]];

That is so important and it is always easy to forget about. Actually there is one way to debug it when the fonts doesn't show up as you expected by using:

NSLog(@"Available fonts: %@", [UIFont familyNames]);

So far, all is good.

But if you use Xcode 4.3.2......

But if you use Xcode 4.3.2, although you are very sure that every steps above are done properly, you still can't find the font you expected in the log message of [UIFont familyNames]. Here is the reason: You have to go to "Copy Bundle Resources" in "Build Phases" of your project, and add font file to that list. Then run the project. Bang! It works now. I've no freaking idea what's going on.It seems there is a bug relating to custom font in Xcode 4.3.2.

IOS使用CAReplicatorLayer重建动态的倒影

2012-03-31T00:00:00+00:00

最近在看ios中关于core animation的一些东西,其中就有一个是任何创建倒影。创建倒影应该是蛮常见的吧，比如你打开iphone中的音乐，这个时候如果你将你的手机横过来，就可以看到这个cover flow的效果了。仔细看的话，你会发现这个每张专辑图片下都有一个倒影。是的，这便是这篇文章的主题。我们会先讲一个普通且常见的创建倒影的方式，然后讲一讲它的缺点，最后讲讲新重建倒影的方式。常见创建倒影的方式：

创建一个origin layer，然后将图片设置为它的contents
重建一个跟origin layer大小一致的reflection layer,将origin layer赋值给reflection layer,并且将其位置定位到origin layer的下面
如果可以，你可以设置一个什么半透明效果layer(optional) 让倒影看起来有点朦胧的感觉
创建一个origin layer的mask,并且是一个带有gradient渐变效果的层，而且它的高度只有图片高度的一半
将reflection layer加入到origin layer的sublayers中

直接上code snippet:

- (void)createReflection {

[self.view setBackgroundColor:[UIColor whiteColor]];
CALayer *contentLayer = [CALayer layer];
UIImage *flag = [UIImage imageNamed:@"chuck-norris.png"];
contentLayer.bounds = CGRectMake(0, 0, flag.size.width, flag.size.height);
contentLayer.contents = (id)flag.CGImage;
contentLayer.position = CGPointMake(self.view.center.x, 10);
contentLayer.anchorPoint = CGPointMake(0.5, 0);
contentLayer.backgroundColor = [UIColor blueColor].CGColor;
contentLayer.cornerRadius = 4.0;

CALayer *reflection = [CALayer layer];
reflection.bounds = contentLayer.bounds;
reflection.position = CGPointMake(contentLayer.bounds.size.width/2, contentLayer.bounds.size.height * 1.5);
reflection.contents = contentLayer.contents;
reflection.transform = CATransform3DMakeRotation(M_PI, 1, 0, 0);

CALayer *blackLayer = [CALayer layer];
blackLayer.backgroundColor = [UIColor blackColor].CGColor;
blackLayer.bounds = reflection.bounds;
blackLayer.position = CGPointMake(blackLayer.bounds.size.width/2, blackLayer.bounds.size.height/2);
blackLayer.opacity = 0.6;
[reflection addSublayer:blackLayer];

CAGradientLayer *mask = [CAGradientLayer layer];
mask.bounds = CGRectMake(0, 0, reflection.bounds.size.width, reflection.bounds.size.height /2 );
mask.position = CGPointMake(mask.bounds.size.width/2, mask.bounds.size.height);
mask.anchorPoint = CGPointMake(0.5, 0);
mask.colors = [NSArray arrayWithObjects:
(id)[UIColor clearColor].CGColor,
(id)[UIColor whiteColor].CGColor, nil
];

reflection.mask = mask;

[contentLayer addSublayer:reflection];

[self.view.layer addSublayer:contentLayer];
}

代码看起来还是蛮直观的，但是这里有几个重要的基本概念得弄清楚。第一个坐标系(Geometry)：

CALayer *contentLayer = [CALayer layer];
UIImage *flag = [UIImage imageNamed:@"chuck-norris.png"];
contentLayer.bounds = CGRectMake(0, 0, flag.size.width, flag.size.height);
contentLayer.contents = (id)flag.CGImage;
contentLayer.position = CGPointMake(self.view.center.x, 10);
contentLayer.anchorPoint = CGPointMake(0.5, 0);
contentLayer.backgroundColor = [UIColor blueColor].CGColor;
contentLayer.cornerRadius = 4.0;

这一段代码很简单就是将contentLayer定位到屏幕水平中间，竖直方向离顶端10个point的地方，然后设置了它的背景颜色和圆角效果（如果要对image做复杂的圆角处理，参考WWDC 2011 Practical Drawing中Bezier Path Drawing). 但是这里却体现core animation中看似最简单实际比较复杂的部分: Geometry. 它有几个常常出现的关键字: position, bounds, anchorpoint, 基本上它们是结对出现的。如果你看过apple的developer documentation中关于"Layer Geometry and Transforms“ 以为自己懂了，然后一看到实际代码却晕了的人，可以看看接下来这段解释。 ios的坐标系是这样: x轴往右是正，y轴往下是正。这根一般的坐标系不太一样。 bounds 很简单，它有origin 和size两个部分组成。由于bounds是针对自己所在的layer,所以它的origin没有啥可以参考的。所以设置bounds基本上主要目的是设置它的高度和宽度。而position不一样，它是针对super layer而言的，这一点跟frame一样。而AnchorPoint却跟bounds和position又不一样，它是相对于bounds和position的比例而言，所以它是小数的方式表示。如果没有显示指明anchorpoint的值，默认是(0.5,0.5)；如果没有显示指明position的值,它的值是(0,0)。这一段描述还是过于抽象，这样从上面的例子来说吧。我们先设置bounds，那就是contentLayer的长和高。然后设置它的position，它是在水平的中间和竖直方向的顶部。假图片的长高是200. 那么这里如果不设置anchorPoint的值，那么它默认是(0.5,0,5) ，那么你会看到图片前（200 - 100 - 10 = 90） 90像素超出了整个屏幕。因为它（0.5,0.5)定义了postion将是在bounds横轴宽的中间位置和数轴的中间位置。如果我们想让图片在10像素时才开始显示，其实也是想让数轴上的position刚刚也是bounds的origin y轴起点，由于y轴往下是正，所以我们就是将anchorpoint设置为(0.5,0) . 那你再看如果设置mask层时，可能就相对容易了。我们先设置它的长高，长度是一样的，但是高度就只有一半了。然后接下来需要注意了： mask.position = CGPointMake(mask.bounds.size.width/2, mask.bounds.size.height); 你需要注意到position并不是CGPointMake(mask.bounds.size.width/2, 0), 因为我们需要做倒影，所以在 reflection.transform = CATransform3DMakeRotation(M_PI, 1, 0, 0); 调用了CATrasnform3D来对空间做出调整。这里我们绕着x轴，顺时针旋转了180度，所以整个reflection层都倒过来了。但是它是按照哪个点进行旋转了,虽然我们知道它是按照x轴旋转的？所以这里的anchorpoint就是重要的，它作为任何改变层的geometry的轴心点。我们知道如果我们不指定anchorpoint，它默认是(0.5,0,5)就是在层的正中间。所以图片才会正确的翻过来。但是因为这种transform赋值是会应用到所有的sublayer 包括自己的。所以此时reflection包括所以它的子层的坐标系发了改变，y轴已经倒了过来，现在是指向了上面。试想一下，x轴不懂，顺时针转180度，此时是不是y轴就刚好倒了它的反面。所这正是mask的position.y 是mask.bounds.size.height而非0的原因。所以这个代码还行，但是它有些缺点，其中最重要的是它是静态的。假设我们此时我们现在在contentLayer上了加了一个文字层，我现在想我点击这个文字增加一个animation动画,在contentlayer上上下活动，然后我想这个倒影层始终保持更新同步（文字也在倒影层中移动）？这个该怎么做了? 所以这是当前做法的局限性，本质上，它是静态的，而且相比接下来的做法，它的性能也大大不如。

具有动态更新能力的CARepliatorLayer

如果有看过wwdc 2011中Session 421 Core Animation Essentials，你可能记得作者简要的描述了一下CAReplicatorLayer的原理，但是可能没有太实际的感受。所以私底下，我找了找，找到了一位大牛Brian M. Coyner的Core Animation Fun House ，这种就有一个关于如何使用CAReplicatorLayer作倒影的例子。虽然看起来很简单，但还是涉及到一些关于core animation很基础很重要的概念。有感兴趣的可以checkout他的项目，然后仔细看看. 首先神马是CAReplicatorLayer:

The CAReplicatorLayer class creates a specified number of copies of its sublayers (the source layer), each copy potentially having geometric, temporal and color transformations applied to it.

简要的说，它自己能够重建包括自己在内的n个copies，这些copies是原layer中的所有sublayers，并且任何对原layer的sublayers设置的transform是可以积累的(accumulative). 基本上这样的一个关系：我们首先会重建一个CAReplicatorLayer实例，作为我们的sourceLayer, 这个sourceLayer我们需要一份copy，那包括自己在内就是2; 所以我们设置了它的instantCount = 2;这个是包括自己在内总共为2. 然后我们将SourceLayer的宽度设置为image的宽度，但是将其高度设置为image.size.height * 1.5；并且在sourcelayer上加上masksToBounds为true的属性，这样一来我们可以保证超出的倒影部分会cut调一半，加上sourceLayer上正常的image，刚刚好组成了我们的完整的倒影。我们sourceLayer的sublayer就是_imageLayer,。但是我们只是单纯设置instantCount = 2的话，那个_imageReplicatorLayer(这个就指代的是copy过来的第一个变量）是会继承sourceLayers中_imagelayer的Geometry,所以它两是重合的。那么我们必须做出transform, CARepliatorLayer有一个属性叫做instantTransform，这个属性指定了除了原来copy之外所有replication layer的trasnform规则，重要的是它是递增的。比如我们这里需要将imageReplicatorLayer应该往下移动image的高度，这一样来可以保证它是刚刚好在原来imagelayer的正下方，就跟倒影一样。但是不一样的地方是：这个复制的imageReplicatorLayer它不是正常的，它是需要倒过来的，所以我们在transform上使用了 transform = CATransform3DScale(transform, 1.0, -1.0, 1.0); 这一个的意思是大小不变，但是y轴倒过来，这个应用到imageReplicatorLayer的坐标系是y轴朝上。这样以来你就不能单纯是向原来一样移动image.height了，因为y轴反了过来，所以你应该是 -2 * image.size.height 这样以来就搞定了。最后我们给它加了一个渐变层，让它看起来更接近倒影的感觉。

[self.view setBackgroundColor:[UIColor whiteColor]];
[[self view] layer].borderColor = [UIColor blueColor].CGColor;
[[self view] layer].borderWidth = 2;

CAReplicatorLayer *layer = [CAReplicatorLayer layer];
[layer setContentsScale:[[UIScreen mainScreen] scale] ];

[layer setBounds:CGRectMake(0, 0, image.size.width, image.size.height * 1.5)];
layer.masksToBounds =  YES;
layer.anchorPoint = CGPointMake(0.5, 0.0);
layer.position = CGPointMake(self.view.frame.size.width/2, 10.0);
layer.borderColor = [UIColor redColor].CGColor;
layer.borderWidth = 0;

layer.instanceCount = 2;

CATransform3D transform = CATransform3DIdentity;

transform = CATransform3DScale(transform, 1.0, -1.0, 1.0);
transform = CATransform3DTranslate(transform, 0, -[image size].height * 2, 1.0);

layer.instanceTransform = transform;

_imageLayer = [CALayer layer];
[_imageLayer setContentsScale:[[UIScreen mainScreen] scale]];
[_imageLayer setContents:(__bridge id)image.CGImage];
[_imageLayer setBounds:CGRectMake(0.0, 0.0, [image size].width, [image size].height)];
[_imageLayer setAnchorPoint:CGPointMake(0, 0)];

[layer addSublayer:_imageLayer];

CAGradientLayer *gradientLayer = [CAGradientLayer layer];
[gradientLayer setColors:[NSArray arrayWithObjects:(__bridge id)[[UIColor whiteColor] colorWithAlphaComponent:0.25].CGColor, [UIColor whiteColor].CGColor, nil]];

[gradientLayer setBounds:CGRectMake(0, 0, layer.frame.size.width, [image size].height * 0.5 + 1.0)];
[gradientLayer setAnchorPoint:CGPointMake(0.5, 0)];
[gradientLayer setPosition:CGPointMake(self.view.frame.size.width/2, image.size.height + 10.0)];
[gradientLayer setZPosition:1];

[gradientLayer setContentsScale:[[UIScreen mainScreen] scale]];

[[[self view] layer] addSublayer:layer];
[[[self view] layer] addSublayer:gradientLayer];

所以了，如果现在你想在图片层上加一个文字，而且想让文字也出现在倒影中，就是非常简单了。因为所有的replicator layer会监视source layer中sublayers的动作变化，一旦它发生变动，所有的replicator layers会对应的进行重画，而且性能看起来还不错。

//Final step to show it dynamic nature
CATextLayer *textLayer = [CATextLayer layer];
[textLayer setContentsScale:[[UIScreen mainScreen] scale] ];
[textLayer setString:@"Chuck Norris"];
[textLayer setFontSize:18];
[textLayer setAlignmentMode:kCAAlignmentCenter];
[textLayer setShadowColor:[UIColor blackColor].CGColor];
[textLayer setShadowOpacity:1.0];
[textLayer setShadowOffset:CGSizeMake(-4, -4)];
[textLayer setBounds:CGRectMake(0, 0, _imageLayer.frame.size.width, 30)];
[textLayer setPosition:CGPointMake(_imageLayer.frame.size.width/2, _imageLayer.frame.size.height - 20)];
//[textLayer setAnchorPoint:CGPointMake(0.5, 0.5)];
[_imageLayer addSublayer:textLayer];
[self.view setUserInteractionEnabled:YES];
[self.view setMultipleTouchEnabled:YES];
[self.view addGestureRecognizer:[[UITapGestureRecognizer alloc] initWithTarget:self action:@selector(animateTextLayer:)]];

- (void)animateTextLayer:(id)animateTextLayer {
CALayer *textLayer = [[_imageLayer sublayers] objectAtIndex:0];
CABasicAnimation *animation = [CABasicAnimation animationWithKeyPath:@"position.y"];
CGFloat endPoint = [textLayer frame].size.height /2;

//textLayer.position.y
[animation setFromValue:[NSNumber numberWithFloat:textLayer.frame.origin.y + endPoint]];
[animation setToValue:[NSNumber numberWithFloat:endPoint]];
[animation setDuration:3.0];
[animation setRepeatCount:MAXFLOAT];
[animation setAutoreverses:YES];

[textLayer addAnimation:animation forKey:nil];

}

非常酷，而且直观。事实上CAReplicatorLayer可以做出很多经验的效果。 joericioppo 有一个结合CAReplicatorLayer和CATransform3D起来的demo，甚是强大，就是这3D中各种轴旋转，直接把我转晕了，但是透过它，你可以更好的理解CAReplicatorLayer以及如何更好的使用CAReplicatorLayer: https://github.com/joericioppo/CAReplicatorLayer_Animation

2012展望

2011-12-29T00:00:00+00:00

2011年马上就过去了. 感激，丰富，新奇，不舍 2012展望练熟吉他读完刚刚买的一些书籍多多转转 ......

墨尔本生活锁记

2011-12-04T00:00:00+00:00

这次来墨尔本,已然是2011年中的第三次了, 显然已经是对墨尔本熟门熟路了, 刚刚接机的时候司机师傅还跟我介绍墨尔本.这次墨尔本之行,除了在工作上完成来之前定下来的目的之外, 在生活上这一次比前两次可要爽多了. 大概因为前两次来这边, 客户他们觉得并没有很好照顾好我们, 而他们觉得在西安时他们被照顾的很好, 所以这一次他们还是非常热情的. 刚刚好,也是时机碰对了,因为墨尔本正在举行19th Australian Corporate Games 比赛, 而客户所在的公司刚刚也报名参加了其中的很多项目, 当然就包括篮球. 因为之前在西安的时候, 我跟他们打过篮球, 他们觉得我这个技术不错, 于是非常强烈的要求我参加, Sam想了一个办法,让我顶替之前不能上场的球员.要是找我要照片地址等等信息,终于成功的注册上了. 我个人觉得也是蛮搞笑的, 我是TW的然后代表他们上场比赛, 有点外援的意思. 是的,这是我很2的入场证明,哈哈于是我跟Sam约好十二点半,他开车来我住的hotel来接我, 然后周六来到了比赛的场馆, 一进去还是相当不错的, 这场地. 不错, 下午我们将会有三场比赛, 每场比赛分上下两半场, 每半场12分钟. 悲剧的是我们等到快比赛时,然后只有四个人来了, 更悲剧的是这个球服上没有任何的号码和名字, Sam带了一根黑笔, 因为他是搞UI的嘛,所以随手一画就帮我们画了个数字,但跟人家其他队伍比起来还真是寒酸阿. 好在在比赛即将开始时,有一哥们赶到了, 凑齐了五个人比赛了. 第一场比赛, 把眼镜一取下来,发现模模糊糊,我才发现我也有一阵子没打球, 不过也没办法, 硬着上, 结果我老被吹走步, 传球都走步, 然后我带球上篮直接人家打我手了,裁判却不吹, 我觉得这裁判很业余. 比较搞笑的是,即使是知道参加比赛, 大家之前都没在一起练习过, 比赛时都是自己搞自己的.囧~ 毫无疑问,我们连输三场, 对方这个头都是1.9几的, 我们这边缺少一个中锋去枪篮板. 不过大家都玩的挺high的,比完之后大家互相在对方球衣上签名. 上个周三Sam提议去吃"泡馍", 我以为这边真有泡馍了,结果大家走之前一看, 我发现他说的泡馍不是我说的, 他说的是"parma". 囧了, 然后大家想这去哪吃, 整来整去的,大家决定去吃意大利的doc pizza. 去了之后,发现那家店就是在一个意大利街区里,里面各种意大利的吃的, 这家批萨店非常火爆. 于是Mike,Simon,Sam骑自行车, 我跟Kalops和heather就做车去的. 店里的菜单各种意大利文字和地图, 英文的菜名也是各种意大利字, 搞的我晕头转向, 然后我就问了下Simon帮忙定了一个. 六个人,六分批萨, 每个人都轮换这分享吃, 味道确实非常不错. 而且这个啤酒很给力,一开始以为没有问题,喝起来很甜, 结果后劲十足, 晚上回去之后感觉就头痛了.吃完正餐,大家点了一些什么点心,都是非常甜的东西. 吃得饱到不行上周四,参加了Agile Melbourne, 来了几位大牛比如写的Joshua Kerievsky和rebeccawb,主要讲了讲关于产品策略和如何在公司中推行敏捷等等. 这周六, Simon提议去这边的一个地质公园: Werribee Gorge State Park. 我其实也是很好奇这边的地址公园会是什么样子, 跟华山有什么区别了.经过将近一个半小时的驾驶,终于来到了这个公园. 进去之后,发现两边的树木很奇怪,他们的皮都剥落了, 非常瘦直, 而且路上人非常少,整个过程中就遇到了六个人. 路上的一个牌子上写着有可能出现鸭嘴兽,我顿时觉得这个很有意思, 因为之前只有在教科书上看到. 但是Simon这个基本上看不到,因为它们生活在非常难以看到的地方. 很奇怪的是这边的基本上没有什么花, 我以为这个时节应该是漫山遍野都是花,结果不是. 而且两旁很多枯死的树木. 走到中间还没有一段是需要沿着绳索攀岩过去的地方, 我觉得很有意思, 但是就是太短了. 快到路程终点时, 看到了Wallaby, 很小很可爱的动物,非常敏觉,听到我们的声音就藏了起来, 但是藏起来的方式就有点像鸵鸟把头埋在沙粒.但是它的伪装技术明显要高超的多,不仔细看,还看不出来. 走下来大概有六公里的山路, 花了将近三个小时, 路上非常崎岖,但是风景非常不错, 这么长的路, 一路上跟simon也聊了很多, 关于这边的地理环境, 甚至讲到了无政府主义跟敏捷的联系等等, 还是非常拓展了我的知识的.

跨域ajax请求

2011-10-30T00:00:00+00:00

之前写的jenkins-dashboard在几个团队中都在用,效果不错, 但是就是搭建起来有点麻烦. 因为它目前工作原理是起一个ruby进程然后每次去轮训jenkins ci上的每个build的状态(通过jenkins暴露的api),然后写到本地的一个html中.接着在写一个主文件比如dashboard.html,它每隔几秒钟去装载刚才ruby生成的html文件,这样就可以在dashboard上及时显示和更新ci的状态.但是问题是这个太重量级了,你还得装ruby环境和几个gem包甚至可能需要rvm,WTF.于是同事准备只用一个html就搞定,就是说每个几秒钟去发ajax请求到jenkins上拿到最新的构建状态,并更新dom结构或者样式, 这样一来用户就可以主需要下个html文件,并配置下ci的地址和想监控的构建就好了.几天后,我问同事怎么样了,他说ajax请求其他域名出错了. 回头我想了想我是忘记这个事情了, 因为浏览器有Same Origin Policy的安全限制. 好比现在: localhost ----ajax request ---> http://ci.jruby.org/view/Ruboto/api/json 不行, 因为虽然ci.jruby.org这个服务器是暴露了它的json/xml的api,但是这个请求不是在localhost下面,所以无法成功.

怎么实现跨域的ajax?

最简单的办法是将本地的服务器作为一个代理,将第三方的ajax请求都转发出去, 这样一来我在客户端,我基本无需改动.但是它的弊处是它扩展性和伸缩性差. 第二种: 之前说过因为同源策略的限制,所以ajax请求无法跨域,但是还是hack的方法. 在浏览器中,对于script这个tag是一个例外,浏览器可以请求其他域的javascript数据源.那这么一来, 我可以在有一个script tag中定义好callback函数:

那怎么让服务器知道它需要组装数据然后调用你在本地服务器定义的callback函数了?(e.g.doSth()). 所以你看到下面的script tag中src属性指定了callback是doSth这个函数引用. 下面是服务器端的大概逻辑:

data=....
request.getParameter('callback');
callback(data)

这个解决方案被称为jsonp(JSON-with-padding),通常被请求的数据都是以json方式来交换.

JSONP-jQuery支持

jQuery自从1.2开始在getJSON()中支持jsonp模式.如果你想请求远程的一个ajax请求,你需要做两件事情:

在url中加上format=[json|xml]和callback=?
定义你的callback函数

这里的callback=? 这个?占位符会在jQuery执行时,替换成匿名函数的引用. 它会首先将匿名函数转换为一个全局函数同时挂到 window对象上. 然后当请求完成,那么它会自动将其删除. 如果你的请求不是跨域的请求,它会将其转换为一个正常的ajax请求. 比如我想请求

 jQuery.getJSON("http://another.domain.com/money?format=json&callback=?", function(data) {
    alert(data.currency + ', ' + data.account);
 });

但是......

到目前为止,jsonp看起来很不错, jquery对它的支持更是不错, 那么.......问题是? 它需要你在第三方的服务器暴露的服务进行修改,以便使其支持正确响应jsonp模式的请求.可以想象,对于another.domain.com/money?format=json这个功能而言,它不能只返回json对象,而是callback(json_data).我们知道很多时候我们没有对第三方服务进行修改的权限.所以这个时候便是YQL来拯救.

YQL

YQL是Yahoo提供的一个第三方query平台和api, 用来抓取第三方的数据信息,而不需要关心第三方的具体协议(REST)和数据格式(RSS,ATOM,XML,JSON, etc). 引用YQL的一个示意图: 有兴趣的可以去YQL Console玩玩. 但是这里我们关注的是YQL它提供JSONP服务支持,也就是说你可以发送一个请求JSON数据的请求同时可以指定你的回调函数,相当于它给所以第三方的服务提供了自动化JSONP服务支持而不需要你对第三方服务做任何的修改,这就使得跨域的ajax修改不依赖于具体的第三方服务端的设置,完全只需要修改前端的javascript代码. yep~ YQL语法的大概格式是:

http://query.yahooapis.com/v1/public?q=[command]&format=json&callback=?

结合jQuery,比如你想拿到http://ci.jruby.org/view/Ruboto/api/json的结果:

select * from json where url ="http://ci.jruby.org/view/Ruboto/api/json"

但是你需要它对应的REST QUERY的地址(web service):

jQuery.getJSON("http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20json%20where%20url%20%3D%22http%3A%2F%2Fci.jruby.org%2Fview%2FRuboto%2Fapi%2Fjson%22&format=json&callback=?",function(data) {
     alert("view name : "+ data.query.results.json.name);
});

实际上你还可以简化一下jQuery和YQL的使用, 比如构造url的过程, 这里james padolsey就写了一个很方便的封装.

Other Cross Domain Strageties

ok,我们讲过跨域ajax请求的两种办法: 设置代理服务器转发, 使用YQL. 但是这两种并没有本质上解决问题,二者都是通过一些独特的方法绕过浏览器对于跨域请求的限制.实际上,在如今mashup急剧增加, 跨域之间的资源共享确实是非常重要, W3C推出了 CORS(Cross-Origin Resource Sharing), 简要的说它是标准XMLHttpRequest(aka.XHR)的扩展,允许浏览器做出不同域之间的请求. 相比以往的普通的XMLHttpRequest对象的工作方式, CORS不同的是它会先向服务器端发出请求,询问是否其他域的请求是否被允许,如果不是,那么就会被拒绝;如果服务端根据预定规则(有点类似于ACL)去检查其权限为允许放行的话,那么它就会亮绿灯允许这个请求继而拿到第三方的数据. 所以你可以看到在服务器端可能有定义如下的权限规则:

   Access-Control-Allow-Origin: http://www.another.domain.com

Nicholas Zakas写了一篇关于CORS for cross-domain Ajax的文章. 但是CORS的未来还不清楚,浏览器的支持也不是非常好,而且要求权限验证逻辑写在服务端,对于OPS的人维护起来也确实是麻烦,特别是你有多个平台需要维护(qa/staging/production). 相比JSONP的方式,二者各有利弊, 还可以参考easyXDM.

引用

JSONP introduction Yahoo! Query Language Screencast: Introducing YQL from YQL Site YQL: An Introduction on Youtube Loading external content with Ajax using jQuery and YQL Cross-domain communications with JSONP, Part 1: Combine JSONP and jQuery to quickly build powerful mashups Cross-domain communications with JSONP, Part 2: Building mashups with JSONP, jQuery, and Yahoo! Query Language

TODO List: Rewrite Jenkins-dashboard using purely html/javascript, no heavy-lifting ruby.

Jasmine测试

2011-10-22T00:00:00+00:00

Jasmine作为一个javascript的测试框架, 借鉴了很多其他测试框架比如RSpec,JSpec的有点,包含了很多优秀的设计思想.结合最近的使用心得,它的主要优点有:

专心做assertion框架, 独立模块化的设计使其非常好的能和其他库进行集成.比如它可以和像提供非常强大健壮的持续集成支持的jsTestDriver(非常专业的non-headless test/spec runner)互相配合相得益彰; 根据jquery相应扩展出的下面会讲到的jasmine-jquery等.
idomatic的语法. Jasmine借鉴了RSpec的语法设计, 如果你是接触过并使用过rspec(当然除了像DHH这种rspec-offender),你发会发现jasmine的语法跟rspec会有惊人的类似,学习使用起来很快就会得心应手,似曾相识.

接下来Jasmine是如何简化javascript的测试了?

 describe("Calculator", function() {
  it("adding algorithm", function() {
     expect(Calculator.add(1, 2)).toEqual(3);
  });
 });

上面是一个简单的测试案例, 简单的测试Calculator的加法逻辑, 但是你看到Jasmine测试的语法确实非常简洁.jasmine这个对于纯javascript的测试,那是得心应手. 但是通常说javascript测试中两个难点:第一个是DOM的交互;第二个是ajax. 与DOM交互测试难处在于: 你需要准备测试对应的fixture也就是html代码片段, 然后让你想测试的javascript在html上执行, 最后在验证DOM上面元素的变化.另外一个方面这些与dom打交道的可能是第三方的库,比如最常见的jquery. 第一个问题其实也可以解决,那便是我们可以使用$('body').append("some test fixtures"), 但它的可用性不好而且最重要的这个字符串型的html片段非常难看,同时你每次都得在测试最后手动删掉这些fixtures.当然你可以将这些fixture写到一些文件之中, 但是你不得不使用ajax去载入文件然后手动的将其append到body后面,代码非常繁琐而且你每次都得记得去DOM中清除掉这些fixtures. 总结处理html fixtures的两个要点是:

如何更加简洁来处理html fixtures的读取加载. 避免需要硬编码的在<body>上面去添加字符串型的fixtures,; 也需要避免使用非常长的使用ajax来加载html fixtures文件的尴尬
在每个测试案例运行之后在DOM文件中清除掉fixtures,这个应该是默认的行为,而不是需要自己手动添加tear down的代码.

另外一个问题便是在assert的时候, 你要判断某一个元素的有没有class? 你可能需要写成:

    expect($("#market-summary").attr("class")).toEqual("no-margin-wrapper");

因为jasmine内置的matcher并没有提供这些, 这也看出来它这些match并不是给断言DOM来设计的. 不过虽然jasmine提供了类似于rspec的个性化matcher的功能,但是这个没有理由让我们重新发明轮子. 这个时候便是Jasmine-jQuery大展伸手的时候了. 它给jasmine提供了两大支持:

a set of custom matchers for jQuery framework

an API for handling HTML fixtures in your specs

第一个它提供了非常强大的个性化matcher, 简化对于DOM条件的断言. 比如上面的判断元素class name, 在集成jasmine-jquery之后, 我可以写成:

   expect($("#market-summary")).toHaveClass("no-margin-wrapper");

这样一来,代码的可读性也跟着大大提供了. 更多的matcher可以参考jasmine-jquery的github主页. 第二是关于怎么简化处理html fixtures的.

 loadFixtures('myfixture.html');

如果你没有个性化fixture加载的地址, jasmine-jquery会默认去找到spec/javascript/fixtures目录下的myfixute.hml文件,本质上还是使用$.ajax去请求这个文件,然后在append到DOM中. 同时它还提供了一套非常完备的API来处理fixture相关的操作. 另外一个提到便是它每次会在测试案例跑完之后清除掉DOM中掉之前加载的fixture, 所以你不需要每次在之后手动加 $("xxxx).remove(). 另外一个便是测试ajax.这里测试分为两种单元测试和集成测试.集成测试也就是真正会touch服务器端,请使用其他工具比如capybara等. 对于javascript单元测试,在测试时我们都必须注意不能真的让它touch到服务器端,也就是不能真正让它发出ajax请求, 如果发出来了,我们也要阻拦住然后替换它(有兴趣的可以看看sinonjs). 你可以使用jasmine有自己一套mock框架spy, 它可以用来mock, stub和fake你想测试协作者的功能. 假设你有一个ajax请求, 在success的回调函数中做很多事情,比如修改DOM阿等等.

var Validation = function(){
   var init = function(){
      $.ajax({
         url: url,
          // ....
         success: function(data) {
           $("#validation-result").text(data);
         }
     });
  }
  return init : init;
});
var Util = function(){
   var sendAjax = function(url, successCallback, errorCallback,...){
      $.ajax({
         url: url,
          // ....
         success: successCallback
     });
  }
  return sendAjax : sendAjax;
});

在对应的测试中,我们就可以使用spy来mock这个ajax请求,使其并不会真得发出ajax.

describe("Validation", function () {
	it("when ajax request succeeded and callback get called", function () {
		loadFixtures('validation.html');
		expect($("#validation-result")).toBeEmpty();
		spyOn($, "ajax");
		Valiation.init();
		expect($.ajax).toHaveBeenCalled();
		expect($.ajax.mostRecentCall.args[0].url).toEqual('some url');
		$.ajax.mostRecentCall.args[0].success("failed");
		expect($("#validation-result")).toHaveText("failed");
	 })
});

相比之下如果我们将send ajax这段boilerplate放在一个单独的模块,而其他引用它的文件的测试就只需要spy这个模块就好了.

var Validation = function(){
   var successCallback = function (data) {
     $("#validation-result").text(data);
   }
   var init = function(){
     Util.sendAjax("some url", "GET", successCallback);
   }
	 return init : init;
 });

以及测试:

describe("Validation", function () {
	it("when ajax request succeeded and callback get called", function () {
		loadFixtures('validation.html');
		expect($("#validation-result")).toBeEmpty();
		spyOn(Util, "sendAjax");
		Valiation.init();
		expect(Util.sendAjax).toHaveBeenCalled();
		expect(Util.sendAjax.mostRecentCall.args[0]).toEqual('some url');
		Util.sendAjax.mostRecentCall.args[1]("failed");
		expect($("#validation-result")).toHaveText("failed");
	 })
});

而接着对于sendAjax的测试就非常简单了, 我们同样可以选择使用spyOn, 也可以使用sinonjs来mock ajax请求.

Jasmine的问题与注意事项:

基于jasmine的机制,它会将所有的引用包括报表包含在一个静态HMTL文件中(SpecRunner.html). 如果你使用jasmine-jquery,在浏览器中执行时如果你使用firebug进行inspect时,你会发现一个tag: <div class='jasmine-fixtures'><div>, 所有的fixtures会默认加载到这个div下,然后测试案例跑完之后,它会去清空这个div的内容.但是问题是如果你有测试案例中代码会对比如<body>等在<div class='jasmine-fixtures'></div>之外的元素进行操作时,它的痕迹是不会在测试案例跑完之后自己清空, 这个时候需要你自己去手动写代码清空, 否则会影响到下面的测试案例.
因为所有的测试在浏览器中都跑一个窗口下, 所以代码跟浏览器相关的操作都需要仔细考虑. 比如你代码中有重新转向另一个url,
```
window.location.href= "www.xxx.com"
```
然后你去assert它当前的url是不是"www.xxx.com".你会发现这个你的jasmine runner的窗口也跟着跳转了, 然后后面测试也无法跑了.如果你理解了jasmine的工作方式之后,你就知道它为什么会这样了. 这个时候要求你需要划分模块了, 将这个转向的逻辑抽出来放在另一个模块中比如叫做NavigationHelper, 然后测试时,你不需要直接是去
```
 expect(window.location.href).toEqual("www.xxx.com");
```
而是可以spy这个NavigationHelper. 这也许是为什么javascript测试覆盖率很难达到100%的一个原因之一.
jasmine在测试全局变量时非常不给力.但是问题不在于jasmine不给力,问题在于为什么引入全局变量. 就像Douglas Crockford说的: global variables are evils. 但是问题是有一些是第三方库强迫的,而且因为商业上原因,你还不能把它去掉或者替换.这个时候你只能采用ad-hoc的方式了.
jasmine有一个有意思的事情便是如果你的spec测试中, 有语法错误. 比如在calculator-spec.js中
```
  expect(1 + 1).toEqual(2);
```
你给粗心写成了
```
  expect(1 + 1).toEqual(2))));
```
然后你去跑jasmine测试, 你会发现spec runner依然是绿色. 但是如果你仔细看, 你会发现这个测试案例的数量却是减少了.在定睛一看,你会发现这个calculator-spec.js文件就没有被执行, 也就是它会忽视整个测试文件,但是它却可以心不惊肉不跳的执行下面的测试. 解决办法: 保持良好的观察力, 养成记住上次测试跑了多少个测试案例的数量, 这当然是开玩笑了. 这种情况下两种办法: 第一种可以使用jslint来进行语法格式检查. 比如这个它会提示:
```
  Lint at line 17 character 20: Expected ')' to have an indentation at 7 instead at 20.
          expect(1 + 1)).toEqual(2);
```
当然一个良好的项目有javascript测试,但是却没有jslint检查,这个就不太专业了. 另外一个办法是:你可以跑javascript coverage, 一个好的项目的话,对于测试覆盖率是有要求的,如果没有达到一定的阈值,就会报错. 比如这里,一个语法问题导致整个测试文件都没跑,通常情况下,测试覆盖虑应该都会降,你这个时候跑CI应该都会错误.但是这个也是不一定的, 它没有jslint检查那么彻底.

引申:

javascript单元测试是非常重要的,它写起来的容易与否跟我们是否合理的划分模块和依赖, 有很大的关系.

就是之前提出的全局变量的事情, 你可以将参数的计算逻辑从第三方库中抽离出来, 这样可以独立测试并覆盖到, 最大限度地做到能测就测到. 另外一个case便是因为javascript测试需要的fixtures并不会随着产品代码的修改而及时更新,但是这个时候你跑jasmine测试它并不会失败.这个事情上次跟gigix聊了一会, 他建议是尽量将javascript划分成比较合理的模块,这样各个模块都可以独立测试,然后通过真正的集成测试比如jasmine来验证行为.比如有一个段逻辑是计算弹出盒子的位置,这个计算位置的逻辑可以抽离出来,然后在写一个cucumber测试,这个测试只要验证盒子出来就好,而不需要关心这个盒子在哪个位置,因为那部分逻辑已经在单元测试中覆盖到. 我后来想到包括你想前面提到的window.href.location这一段逻辑,它抽离出来的好处在于在集成测试上面,哪怕有非常多的地方掉了这个逻辑, 但是我只需要写一个覆盖到重定位的cucumber场景就好了. 其实我想了想这个问题是: 究竟得写多少cucumber测试才能确保产品html的改动同时及时反应在javascript的单元测试中fixtures上面,使得二者保持同步,确保javascript代码时刻能正确工作.之前说到jasmine的设计非常巧妙, 其中之一便是体现在跟CI的集成上面. 简要的来说现在的spec runner的host分为两种: headless>和non-headless. 第一种就是不起显示的浏览器,也就是没有浏览器, 你可以使用java实现的js引擎rhino以及在模拟浏览器中DOM实现的Envjs,优势是速度上非常快.第二种使用浏览器的比如jsTestDriver, jasmine有跟其集成的jasmine-jstd-adapter . 参考: 想简单体验一下: http://try-jasmine.heroku.com/ railscast上面的介绍: http://railscasts.com/episodes/261-testing-javascript-with-jasmine jasmine-jquery作者的博客: http://testdrivenwebsites.com/2010/07/29/html-fixtures-in-jasmine-using-jasmine-jquery/ SinonJS: Planning, Cheating and Faking Your Way Through JavaScript Tests

古今笑

2011-09-24T00:00:00+00:00

最近读了节选部分的<古今笑>,清李渔给古今笑做的序有提到这本书"雅俗并嗜,购之唯恨不早".<古今谭概>是冯梦龙收集的一些稗官野史中的记录,然后加入自己的一些品评,跟苏轼的有点类似.这类的笔记有意思的在于它提供了很多侧面的方式帮助我们了解古人的形象和生活,拉近了距离. 异服翟耆年好奇，巾服一如唐人，自名唐装。一日往见许彦周。彦周髽髻，著犊鼻裤，蹑高屐出迎。翟愕然。彦周徐曰：“吾晋装也，公何怪？” 梦龙注: 只容得你唐装！ ps:有点像司马相如跟卓文君私奔时"自着犊鼻裤，与保庸杂作，涤器于市中"的风范. 不知道这个"犊鼻裤"是啥样?看到了这篇"[二十年二十题之]犊鼻裤 " ,难道就是日本相扑中穿得那个丁字裤? oh no. 浴酒石裕造酒数斛，忽解衣入其中，恣沐浴而出，告子弟曰：“吾平生饮酒，恨毛发未识其味。今日聊以设之，庶无厚薄。” ps:喝酒能喝道这个地步,叫我等只能喝两瓶啤酒的人情何以堪?

痴趣
陶渊明日用铜钵煮粥为食，遇发火，则再拜曰：“非有是火，何以充腹？”
贾岛常以岁除，取一年所得诗，祭以酒，曰：“劳吾精神，以是补之。”
韩退之尝登华山巅，穷极幽险，心悸目眩，不能下，发狂号哭，投书与家人别。华阴令百计取之，方能下。
张旭大醉，以头濡墨而书。
ps:每当看到最后一句, "狂草"张旭这个举动,令人感叹,这个是痴迷到了何种地址,都叫人羡慕.
我们应该效仿贾岛,过年时,取一年所写代码,祭以酒，曰：“劳吾精神，以是补之。”

拙对
《谐史》：河南一士夫延师教子，其子不慧。出对曰：“门前绿水流将去。”
子对云：“屋里青山跳出来。”士夫甚怒。
一日士夫偕馆宾诣一道观拜客。道士有号彭青山者，脚跛，闻士夫至，跳出相迎。
馆宾谓士夫曰：“昨令公子所谓‘屋里青山跳出来’，信有之矣。”士夫乃大笑。

ps:这学问可比还珠格格好多了

不知杜少陵
宋乾道间，林谦之为司业，与正字彭仲举游天竺小饮。
论诗，谈到少陵妙处，仲举微醉，忽大呼曰；“杜少陵可杀！”
有俗子在邻壁，闻之，遍告人曰；“有一怪事，林司业与彭正字在天竺谋杀人。”
或问：“所谋杀为谁？”曰：“杜少陵也，但不知何处人。”闻者绝倒。

雪诗

陆诗伯《雪》诗云：“大雪洋洋下，柴米都长价。板凳当柴烧，吓得床儿怕。”又云：“玉皇大帝卖私盐，一个苏州拖面煎。”
又云：“不闻天上打罗橱，满地纷纷都是面。”又云：“昨夜玉皇哀诏到，万里江山都带孝。”

ps:很好的描述当前的物价水平下人民大众的生活

点金成铁
梁王籍诗云：“蝉噪林愈静，鸟鸣山更幽。”王荆公改用其句曰：“一鸟不鸣山更幽。”
山谷笑曰：“此‘点金成铁’手也！”

ps:王安石悲剧了,这一改可是没有"绿"字那么传神

马上食饼
张衡由令史至三品，已团甲，退朝，于路傍见蒸饼新熟，遂买得，于马上食之。为御史弹奏，竟落甲。
梦龙注:向闻二卵弃将，今见一饼失官，若在晋人，反为任诞。

 ps:难得真性情 可以在唐代,而非晋朝

鸣鹅
会稽有姥，养一鹅，善鸣。右军求市不得，遂携亲友就观。姥闻羲之至，烹鹅以待。右军叹惜弥日。
ps:王羲之果然爱鹅,难道鹅跟书法之间有联系?

爱杜甫、贾浪仙诗
张籍取杜甫诗一帙，焚取灰烬，副以膏蜜，顿饮之，曰：“令吾肝肠从此改易。”
李洞慕贾浪仙诗，铸铜像，事之如神，常念贾岛佛。

ps:张籍中晚年的诗句还真有杜甫的现实注意风范;

石
梦龙注:袁石公曰：“陶之菊，林之梅，米之石。非爱菊、梅与石也。吾爱吾也。”
僧敄周有端州石，屹起成山，其麓受水可磨。米后得之，抱之眠三日。
ps:看到米芾看完<灌篮高手>之后的情形,抱着篮球睡

柳三变
柳耆卿为屯田员外郎，初名三变。自作词云：“才子词人，自是白衣卿相。”
后有荐于朝者，仁宗曰：“此人风前月下且去填词！”
由是不得志，无复检率，自称“奉圣旨填词柳三变”。
梦龙注:按柳永死日，家无余财，群妓合金葬之郊外，每春月上冢，谓之“吊柳七”。 子犹曰：“生虽白衣贱，死得红裙怜。北邙冢累累，白杨风满天。卿相代有作，谁复追黄泉。呜乎柳三变，风流至今传。”
ps:霸气外漏

肉唾壶
苻朗尝与朝士宴。时贤并用唾壶。朗欲夸之，使小儿跪而张口，唾而含出。
南宋谢景仁裕性整洁。每唾，辄唾左右人衣。事毕，即听一日浣濯。每欲唾，左右争来受之。
严世蕃吐唾，皆美婢以口承之。方发声，婢口已巧就。谓曰“香唾盂”。

张鹭鹚
开宝中，神泉县令张某，外廉而内实贪。一日自榜县门云：“某月某日是知县生日。告示门内典级诸色人，不得辄有献送。”
有一曹吏与众议曰：“宰君明言生日，意令我辈知也。言不得献送，是谦也。”众曰：“然。”
至日各持缣献之，命曰“寿衣”。宰一无所拒，感领而已。复告之曰：“后月某日，是县君生日，更莫将来。”无不嗤者。
众进士以鹭鹚诗讽之云：“飞来疑似鹤，下处却寻鱼。”
ps:跟目前官场一样, 古今都一样

食人胆
五代赵思绾反。尝言“食人胆至千，刚勇无敌”，每杀人，辄取胆以酒吞之。后为郭从义所擒。
生食人耳
宋王彦升俘获胡人，置酒宴饮，以手裂其耳，咀嚼久之，徐引卮酒。俘者流血被面，痛楚叫号，彦升谈笑自如
ps:如此悍将给赵匡胤守西北,不愧是心腹,但是这个太恐怖了把

程师孟、张安国
程师孟尝请于荆公曰：“公文章命世，某幸与公同时，愿得公为墓志，庶传不朽。”公问：“先正何官？”
程曰：“非也。某恐不得常侍左右，预求以俟异日。”
又王雱死，张安国披发籍草，突于柩前，曰：“公不幸未有子，今夫人有娠，某愿死，托生为公嗣。”
京师嘲曰：“程师孟生求速死，张安国死愿托生。”

 ps:难道此张安国正是被辛弃疾五十骑于万军中擒拿到的叛将?

不敢须
少司徒王祐谄事太监王振。振一日问曰：“王侍郎何故无须？”曰：“老爷无须，儿子岂敢有须？”

车武子妇
车武子妇妒。武子偶偕妇兄夜归，留宿外馆，取一绛裙挂屏上。妇出窥，疑有所私，拔刀径上床，发被，乃其兄也，惭而退。
ps:车胤(也就是车胤囊萤夜读)典故来由之人,居然有如此悍妇, 不得不唏嘘,不过也许他说不定是smart gay

头断复连
正德时，济下一秀才遭流贼乱，奔避不及被贼砍，觉头落胸间而喉不断，亟以手捧头置之项上，热血凝结，痛极遂死。
久之稍苏，卧野田间。寇退，家人求尸舁归。旬日不死，颇能咽汤粥。百日痂脱，视其颈瘢痕如絙入腮下。
ps:坑爹阿,不可能把.不过说不定人的求生意识也有可能

爱东坡
陆宅之善谐谑，每语人曰：“吾甚爱东坡。”时有问之者，曰：“东坡有文，有赋，有诗，有字，有东坡巾。君所爱何居？”
陆曰：“吾甚爱一味东坡肉！”闻者大笑。
ps:东坡要是生在现在,恐怕也只能望猪肉兴叹

解缙
解缙尝从游内苑。上登桥，间缙：“当作何语？”对曰：“此谓‘一步高一步’。”及下桥，又问之。
对曰：“此谓‘后边又高似前边’。”上大悦。一日，上谓缙曰：“卿知宫中夜来有喜乎？可作一诗。”
方吟曰：“君王昨夜降金龙。”上遽曰：“是女儿。”即应曰：“化作嫦娥下九重。”上曰：“已死矣！”
又应曰：“料是世间留不住。”上曰：“投之水矣。“又应：”翻身跳入水晶宫。”上本欲诡言以困之，既得诗，深叹其敏。

解缙四岁出游市中，偶跌，众笑之。吟曰：“细雨落绸缪，砖街滑似油。凤凰跌在地，笑杀一群牛。”

ps:这可可以跟纪晓岚有得一拼,不愧是明朝第一位大三元的人,才思敏捷, <永乐大典>的编辑者.

公猴
三杨当国时，有一妓名齐雅秀，性极巧慧。一日命佐酒。众谓曰：“汝能使三阁老笑乎？”对曰：“我一入便令笑也。”
乃进见。问：“何来迟？”对曰：“看书。”问：“何书？”对曰：“《烈女传》。”三阁老大笑，曰；“母狗无礼！”
即答曰：“我是母狗，各位是公猴。”一时京中大传其妙。

ps:妙不可言~  "公猴" 谐音似 "公侯", 三杨都是公的爵位,这么位高权重的人,这么有名的三杨,却被一妓女调侃.佩服

苏小妹
东坡有小妹，善词赋，敏慧多辩，其额广而如凸。东坡尝戏之曰：“莲步未离香阁下，梅妆先露画屏前。”
妹即应声云：“欲扣齿牙无觅处，忽闻毛里有声传。”以坡公多须髯，遂亦戏答。时年十岁耳。

李东阳
焦阁老芳，面黑而长，如驴。尝谓西涯曰：“君善相，烦一看。
”李久之乃曰：“左相象马尚书，右相象卢侍郎，必至此地位。”“马”与“卢”合，乃一“驴”宇，始知其戏。

邃翁冬天气盛，而西涯怯寒。二公同坐，西涯屡以足顿地作声。
邃翁曰：“地冻马蹄声得得。”西涯见其吐气如蒸，戏云：“天寒驴嘴气腾腾。”

李西涯居政府时，庶吉士进谒，有言“阁下李先生”者。公闻之，既相见，因曰；“请诸君属一对，云‘庭前花始放’。
”众疑其太易，转思未工。各沉吟间，公曰：“何不对‘阁下李先生’？”相赞而笑。

ps:李东阳这位首辅果然是才气外漏,有意思,搞笑. 而杨一清跟李东阳两位明代诗歌的先驱者,私底下如此搞笑,读起来倍感亲切.

石学士
石曼卿尝出游报宁寺，驭者失控。马惊走，曼卿堕地，戏曰：“幸是石学士，若瓦学士，岂不破碎！”
ps: "天若有情天亦老，月如无恨月长圆" 对得起下一句的石学士生活中也是宽容大气. 人品,才气哪一样能不让欧阳修,梅尧臣等诚服

十七字诗
正德间，有无赖子好作十七字诗，触目成咏。时天旱，府守祈雨未诚，神无感应。
其人作诗嘲之曰：“太守出祷雨，万民皆喜悦。昨夜推窗看，见月！”
守知，令人捕至，曰：“汝善作十七字诗耶？试再吟之，佳则释尔。”
即以别号“西坡”命题。其人应声曰：“古人号东坡，今人号西坡。若将两人较，差多。”守大怒，责之十八。
其人又吟曰：“作诗十七字，被责一十八。若上万言书，打杀！”守亦哂而逐之。

一说：守坐以诽谤律，发配郧阳。其母舅送之，相持而泣。
泣止，曰：“吾又有诗矣：发配在郧阳，见舅如见娘。两人齐下泪，三行。”盖舅乃眇一目者也。
ps:有创意

有兴趣者可以去看看<古今谭概>这本书

近期郑大晔校总结

2011-09-17T00:00:00+00:00

这两期的郑大晔校,感觉收获很多.第一次讨论一个程序员应该具备的好的编程习惯;第二次像经验丰富的资深程序员发问.我总结了一下郑大晔校中自己感兴趣的话题和近期的工作碰到问题:

1.熟悉编辑器的快捷键.

这一点我去年进公司时印象深刻,那个时候是一个.NET项目使用Visual Studio,跟胡凯结对编程的时候,胡所一般都会把鼠标收起来,完全丢弃, 直接各种键盘流, 看得我惊叹.紧接着胡所就把Resharper的快捷键打印出来,让大家私底下回去记忆,顺便讲到了他记忆快捷键的方法(使用小卡片,在公交车上背阿). 这个的好处就是使你的大脑思维集中在你关注的事情上面,保持自己的编码节奏,提高开发的效率. 所以熟悉快捷键操作是非常重要的,公司里几乎每个人接触到新的编辑器,都会从先熟悉快捷键开始.

2.Tasking任务划分.

这里的Task指的不是评估story,而是程序员在编码中将story拆分成一个个小任务,这些任务可以以通过对应的测试为验收标准.之前徐昊去年在做OO bootcamp时,我记得印象深刻的是他强调了划分小任务的重要性.他问道读过<<Test Driven Development by exmaple>>的觉得里面重要的是啥? 最重要的是他每隔小节中都会提到的任务列表, 每次做完一个小任务,他会划掉,并且加上新的小任务,然后用TDD方式实现.同时另外一个感触是每次团队中有新人加入,我们的结对编程会是,有经验的人会跟新手一起过一遍story,然后划分为一个个的小任务列表,到后来就是直接让新手自己先写出任务表, 写完之后一起浏览过一遍, 这样做非常好在于当你在写task的时候, 你会发现很多时候你的问题的认识是不到位的.Tasking好处在于帮助大脑将复杂的事情拆分为简单可行的任务列表.任务的划分可以使用垂直划分的方式.

3. Timebox.

这个Timbox不是说到Scrum中的迭代,而是在编码过程中对于某件事情固定一个时间范围约束.为什么会这样?我本人就犯错的经历.某次开发中接到一个技术任务,而后按照我认为我自己的理解开始干活,这个过程中碰到问题了,那好我就潜进去看为什么,结果吭哧吭哧搞了一天,最后发结果发出去了,人家说不对,你理解错了,结果就整个浪费了一天.我在想接到这件事情时,我自己很清楚这个东西应该只需要2个小时或者做多半天的.但是中间自己遇到问题却也没有及时跳出来问问题,自己仍在钻研,结果半天之后发现自己的理解错了.所以很多时候遇到事情,可以在开始干之前给自己一个时间范围,超过这个时间了,没有完成,你可以暂时停下来跟其他人交流确认你遇到的问题,毕竟自己一开始按照自己理解而设置的时间阈值不够,也许真是问题比自己了解的要复杂,或许其他时候是自己的理解不准确或者方向不正确.比如上次碰到的Apache SSL的配置问题, 因为大家都不是非常熟悉Apache的一些配置,吭哧吭哧在那里搞了半天,结果还是没有搞出来.后来跳出团队,问其他团队Apache经验比较的多的人,一解释立马就豁然开朗了.

4.观察力.

公司很多项目中有用到故事墙来描述故事的状态,哪些是在In Dev,那些在In Dev Done, 哪些是In QA;也有大的电视投放jenkins上构建的状态(见前一篇博客);这些都是为了将项目的状态尽量的视觉化出来,让大家都在同一个上下文中.我注意到胡所貌似给admin也有应用类似的理念,公司入门口就有一个白板,上面标注着admin哪些工作正在做,哪些做完了等等, 这样即使是开发写代码的人也知道admin在做什么事情.比如admin谈到对技术不是特别懂,可能有时候比如招聘会有人问技术之类的,就不太好回答, 那我一个同事就有给admin做一个讲座,主要介绍当前办公室正在使用的技术(以至于有HR直接转成BA).目前有很热的为了消除开发人员与运维人员之间隔阂障碍的DevOps, 我觉得目前我们就有做DevAdmin, 这个就像设计模式源于建筑设计师Christopher Alexander.虽然之前写得jenkins-dashboard写的很简单还有很多不足,但是看到有其他项目也有用到,效果还不错,这个也挺满足的.做一个有心人,观察生活或者工作中其他人是否有碰到不便或者麻烦的事情,想想以自己的知识和能力能否帮助到别人, 这是非常重要的.

5.结对编程.

结对编程这件事情其实源自三哥的一个提问, 其实也是我自己的一个疑惑: 是否什么情况下都能结对?有什么情况是不适合结对? 其实反问自己, 尽管每天都在结对编程中, 但是对于最基本的问题,我却也是无法回答.但是近期看到前TWer Jay Fields的最新更新的文章Life after Pair Programming, 结合他自己实际一一"驳斥"了 Pair Programming Benefits中提到的pair的优点,给人满多思考的.

6.站立会议.

最近的站立会议中我们发现客户的Product Owner,在我们这边发言时,不知道什么原因,可能是技术方面的东东她不太感冒,扭过头去跟旁边的UI设计人员讨论mockup去了.站立会议完了之后,同事提问到为什么product owner会提不起精神,我们要思考我们standup发的言,对于product owner或者说iteration manager有意义才行. 但是之后,我自己很有疑问: 我们是开发人员,更多的时候确实说的事情在商业价值不是非常明显能表达出来. 于是看了看It's Not Just Standing Up: Patterns for Daily Standup Meetings. 作者提到站立会议应该说: 昨天做了什么? 今天将要做什么? 有什么东西阻碍我们的进展? 并且提到不太好的站立会议的一个特征便是: Reporting To The Leader. 更多的时候,站立会议的意义在于让团队知道对方在做什么,目前团队遇到的问题,使得大家能在同一个上下文中.

Jenkins的一些实践

2011-09-11T00:00:00+00:00

持续集成已经是大家都非常熟悉的概念了,可以参考Martin的文章Continuous Integration .而Jenkins(原名hudson)正是业界最为流行的持续集成引擎.本文正是总结了近期工作之中一些使用jenkins的收获.

Keep the Build Fast

保持CI(持续集成服务器)上的工作快速是一个非常重要的方面, 这样才能提供快速的反馈.没有人会忍受一个构建跑上一个小时,毕竟喝上一个小时的茶或者咖啡也还是真是蛮浪费时间的. 之前的持续集成构建都得跑上半个小时,因为这个构建的代码库非常大,不止一个项目.每次构建都会按照一下顺序逐一执行: 可以看出这是一个具有完整构建阶段的流程. 但是每一个阶段都是按照顺序执行,是不是它们之间有严格的依赖了? 比如跑javascript单元测试得先跑quality检查了? Quality Check大致使用metric_fu等质量检测的工具来进行,包括检查代码复杂度,重复度, 以及jslint语法的检查等等. 这就跟javascript单元测试这个阶段没有直接的耦合.同样的也可以对其他阶段进行分析. 所以能否将每个阶段都抽取出来了作为一个独立的构建, 然后触发时每个构建都同时执行, 而不是顺序执行了? 这里我们可以使用Jenkins的一个插件Join.

This plugin allows a job to be run after all the immediate downstream jobs have completed. In this way, the execution can branch out and perform many steps in parallel, and then run a final aggregation step just once after all the parallel work is finished.

Join适合于"钻石"依赖型的项目. 也就是我们上面的构建流程可以转换成: 可以看到上面的几个阶段之间没有什么依赖,同时它们却都有一个聚合的阶段,便是Build Artifact. 这样的好处在于位于钻石形状上方的构建job是可以并发的执行,相比顺序执行,时间可是节省不少. 对于一个代码库融合了多个项目的情况, 另外一件可以做得事情是, 对于每个特定的阶段按照不同项目或团队划分.比如cucumber构建,它可能是包括好几个项目的features.所以每一次测试失败, 首先是难以知道是哪个项目的feature挂了, 另外一个也是非常慢. 所以可以做得是将之前单独的一个cucumber构建, 按照不同项目进行划分, 比如projectA-cucumber , projectB-cucumber 等等. 对于这种情形,另外一个可以追踪的问题便是能不能把不同项目从同一代码库中抽离出来放在独立的代码库中, 或许上面提到的问题都不存在了.

Everyone can see what's happening

当每次准备好一个Jenkins,看到的构建状态与小球颜色的关系总是让人有点不习惯.为什么? 因为它用蓝色代表成功, 你可能想根据测试驱动开发我们熟悉的节奏"red-refactor-green", 如果构建成功的话,应该是绿色阿.因为蓝色更容易被色盲的人识别.所以如果你们项目没有色盲的人,就非常推荐使用GreenBall这个插件, 这样一来所有构建成功都会显示绿色. 对于团队中的个人而言,每次都得打开Jenkins的地址去查看构建状态,这个有时候太麻烦. 所以你可以尝试使用CCMenu,配合Growl, 你可以时时看到构建的状态. 对于个人,CCMenu可能已经满足了大部分需求. 但是如果团队稍微大一点, 比如4-5个人以上, 同时在一块地方,有没有更好的方式来显示构建的状态了? 我们通常会是有一个独立的显示器,放在大家抬头就能看到的位置, 这个显示器主要是显示构建的状态. 当然你可以是在显示器中打开Jenkins的地址,让大家自己查找想看的构建信息,但是就是噪音太大. 所以你可以尝试使用Jenkins的一个插件: Radiator View Plugin. 你可以增加一个View,然后选择Radiator View,并配置你想监控构建的一些信息. 这个看起来还满不错的, 而且配置简单. 在工作中,客户有好几个专门的显示器,用来显示每个构建的状态.类似于下图, 它是用python写的, 因为我也没学过python,借鉴了用户的UI设计, 就用ruby重写了一下.它每一个构建是一个小方块, 绿色表示构建成功, 红色表示失败, 蓝色表示正在构建,同时在右下端有时间戳表示当前构建的时间.这个项目正在开发,应该马上就会放到github上面.你只要给定jenkins的地址,以及你想关注的job的名字就可以了.这个可以参考jenkins-dashboard项目. 是的, 除了这个之外, 还有高级的一点的The Hudson Bear Lamps. 用灯的颜色来表示当前构建的状态. 这个比上面的来得更加直观. 但是这个灯不知道国内能否买到. 还有一点从工作中学到的便是,对于大型的而且是分布式的团队而言, 如何让信息交流更通畅. 对于构建而言, 如果某一次提交将构建弄费了, 首先那个弄费的人(当然不一定是那个弄费的人在修构建)赶紧修改代码让构建通过, 同时还得让其他人,那些跟你不在一地的人知道当前失败的构建有人在处理. 在项目实践中, 就有这样的构建,如果某人将构建弄费,他(她)可以通过加bookmarklet小书签的方式来声明"是我将构建弄费,我正在修理中....", 这个小书签就是发出一个ajax请求,将当前声明人的名字,哪一个构建等信息发给服务器,服务器在修改html的结构在对应的构建上打上某人的名字的邮戳. 当然这个问题我是有疑问的, 理论上如果某一次你将构建搞失败了,第一件事应该是在CI上反转你的提交,这样别人就可以提交,而不必等你修理构建之后才能提交,这个实在是浪费别人的时间.当然这个前提上,大家遵守约定,就是必须是在构建为绿色的情况下才提交,不能说在构建失败后还继续提交.

第一个RubyGem jenkins-remote-api

2011-09-03T00:00:00+00:00

这个Gem主要是用来消化Jenkins提供的API,方便查询CI的状态. 这个RubyGem开发主要用到了bundler,而非jeweler,同时用到了Aruba提供了CLI支持,这样方便了用户使用. 更多信息,参考Github地址: https://github.com/tuo/jenkins-remote-api/blob/master/README.md RubyGem上地址: https://rubygems.org/gems/jenkins-remote-api Developing a RubyGem using Bundler

selenium rc和webdriver

2011-08-17T00:00:00+00:00

前两天在跟结对的同事查看一段我之前写的自动化搭建初始数据的代码时,同事问我为什么需要起一个selenium server才能跑起来,而我们的本地跑的cucumber测试却没有需要你显示的跑一个selenium server.我一时其实也回答不上来,下班后仔细查了查selenium的文档,才豁然开朗.我之前用的是selenium-client, 因为我之前想用它来截取每个浏览器的屏幕, 而selenium-client提供了很好API来跟浏览器打交道(甚至内置的截图方法), 它是基于Selenium RC的.讲到这个先简要介绍一下selenium,它主要用来驱动和跟浏览器做交互,实现web应用程序的自动化测试. 所以意味着selenium必须有一套自己跟浏览器交互的API. selenium根据实现形式的不同分为两种: selenium rc 和 webdriver.

Selenium RC

selenium rc 是比较早提出来的, 它的大概流程是这样: selenium rc客户端先跟服务器端建立联系, 然后服务器端启动一个浏览器,将主要selenium核心的javascript注入到浏览器中. 然后客户端将不同语言编写的代码比如在selenium-client ruby代码是这样

    require "rubygems"
    gem "selenium-client", "~=1.2.16"
    require "selenium/client"
    begin
      @browser = Selenium::Client::Driver.new 
          :host => "localhost",
          :port => 4444,
          :browser => "*firefox",
          :url => "http://www.google.com",
          :timeout_in_second => 60

      @browser.start_new_browser_session
        @browser.open "/"
        @browser.type "q", "Selenium seleniumhq.org"
        @browser.click "btnG", :wait_for => :page
        puts @browser.text?("seleniumhq.org")
    ensure
      @browser.close_current_browser_session
    end

转换为一种中间的指令Selenese,然后发给服务器端,服务器在浏览器中请求对应的URL,拿到页面后,再将Selenese转换成对应的javascript指令注射到浏览器当前页面中并执行,最后在关闭浏览器. 所以selenium RC是使用javascript来跟浏览器进行交互的.

Selenium Webdriver

selenium webdriver采取的一条不同方式,它不是使用javascript来跟浏览器打交道, 而是实现了跟每一个不同浏览器特定相关的原生API来跟浏览器打交道.所以意味着它正对某种浏览器,都有一个对应的driver. 好处在于它绕过了selenium rc使用javascript跟浏览器交互的一些弊病,比如same origin. 同时Selenium Webdriver旨在提供一套更加面向对象方式的API.因为它使用每个浏览器原生API来打交道,所以一方面它比selenium rc会更快更稳定,同时就意味着我们可以这么说"如果你使用的是selenium webdriver的话,你就可以直接抛弃selenium server." 因为你根本不需要启动一个服务器来处理跟浏览器交互.

require 'rubygems'
require 'selenium-webdriver'

driver = Selenium::WebDriver.for :firefox
driver.get "http://google.com"

element = driver.find_element :name => "q"
element.send_keys "Selenium seleniumhq.org"
element.submit

puts "Page title is #{driver.title}"

wait = Selenium::WebDriver::Wait.new(:timeout => 10)
wait.until { driver.title.downcase.start_with? "Selenium seleniumhq.org" }
puts "Page title is #{driver.title}"
driver.quit

之前提到每个浏览器都得有自己对应的driver,现在目前有支持:FirefoxDriver, ChromeDriver, IEDriver和OperaDriver,甚至有IPhoneDriver. 但是相对selenium rc, webdriver它对alert/confirm框的支持还不是很好. Capybara的selenium使用的是哪一种了 selenium RC 或 Webdriver? 你可以从Capybara的github wiki说明上找到: At the moment, Capybara supports Selenium 2.0 (Webdriver), not Selenium RC. 在capybara中,你可以注册不同浏览器对应的driver:

 Capybara.register_driver :selenium do |app|
   Capybara::Selenium::Driver.new(app, :browser => :chrome) 
 end

Ruby Bindings

selenium webdriver 提供了非常好的ruby支持,可以参考Selenium-webdriver Ruby.

References

For more information, pls refer to site docs: http://seleniumhq.org/docs/ Official Blog:http://seleniumhq.wordpress.com/

安装Puppet

2011-07-16T00:00:00+00:00

近些年来出现了很多热词,其中有一个就是Infrastructure As Code,基本的想法是相比原来系统管理员手工运行什么ssh或者命令来管理和搭建不同的机子或者平台, 能否使用一种更加抽象级别的语言来描述这个过程, 将系统底层的东西隔离开来, 使他们能脱离出系统底层各种细节,更多关注自己想到做得事情. 所以就出现这样一些工具: puppet, chef等等. 上周胡凯要我尝试给ThoughtWorks西安也搭建这样的一个平台,我觉得这个东西还蛮有意思的,尝试了下puppet,中间还是遇到了不少问题,不过最终都有调试通过.这篇文章将会简要的介绍怎么安装puppet和中间可能遇到的问题和解决办法. 我的环境是puppet agent使用的是Ubuntu虚拟机(这样比较稳妥), 同时puppet master也用的是Ubuntu. 第一步:安装. 在server上首先你的装上ruby, 然后安装puppet. 在Ubuntu上最简单的安装方式是apt拉, 你可以运行"sudo apt-get install puppet puppetmaster facter", 如果你现在运行puppet -V查看版本的话, 却发现它的版本是"0.25.0",然而主流的版本是"2.6.8".解决办法是: 修改你的/etc/apt/sources.list 加入运行aptitude update (是为了装上最新版本的puppet master 2.6.8, 如果你不修改这一行你装的就是0.25这个版本, 问题很多).

 
deb http://backports.debian.org/debian-backports lenny-backports main contrib non-free
aptitude update
aptitude show puppet

这下你运行puppet -V应该可以看到你的版本已经是最近的版本. 默认这个puppet会安装到/etc/puppet下面.这个时候你需要产生一个puppet.conf文件,木有这个文件,puppet master是启动不了的.你可以使用"puppet master --genconfig > puppet.conf" 来产生这个配置文件. 然后你可以尝试启动puppet master,用如下命令"puppet master --debug --no-daemonize" 你应该可以看到"notice: Starting Puppet master version 2.6.8" --verbose表示的是这个日志输出的详细程度, --no-daemonize表示这个线程将不会转入后台, 你可以在console当中看到日志的输出. 如果出现Could not run: Could not create PID file: /var/lib/puppet/run/master.pid, 就意味着很有可能你上次使用了daemonize的方式启动的, 所以现在这个进程还在后台, 你可以ps -ax | grep puppet ,然后杀死这个进程,在输入命令试试. 紧接着就是puppet agent了,也就是要配置客户端了.具体过程可以参考下面链接:Installation Guide. 安装完成之后可以运行: puppet agent --server=puppet --no-daemonize --verbose那这个--server指的就是puppet master所在主机的地址.极力推荐使用域名而非简单的IP地址. 你也可以在/etc/hosts中加入如下映射:

 
10.29.2.50      puppet

然后在你的puppet master上配置:(这里的10.29.2.5也就是clark-laptop的ip) Ok,搞定. 运行puppet agent --server=puppet --no-daemonize --verbose, 就开始了puppet agent和puppet master之间的ssl过程, 这个过程是可能出现问题较多的地方.这个时候puppet agent会想server端发出确认certification的请求,所以在server端你可以运行" puppet cert --list" 来查看有多少agent的认证请求, 比如这里就有如下输出:

 
clarht-laptop

上面代表的是agent所在主机的域名了,那这个域名是在puppet master主机上/etc/hosts中根据请求过来的agent的ip而对应找到的域名. 然后puppet cert --sign clarht-laptop就可以确认认证了,握手完成. 如果这个过程中,你有碰到" err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key" 这个时候表示你可能将hostname, ip, domain之间的设置给搞错了. 首先确保你的puppet server的映射木有出错.在你的puppet master所在的机子上, 运行"less /ect/resolv.conf" 查看到你的domain name. 然后在你的/etc/hosts中加入

 
10.29.2.50 puppet.corporate.thoughtworks.com puppet

也就是按照"ip <hostname>.<domain name> <hostname>"的方式. 对于agent的配置比较麻烦一点, 首先上到puppet agent所在机子上,命令行中调用"hostname", 获取到agent所在主机的hostname,比如我这里就是"clark-laptop", 同时less /etc/resolv.conf 查看agent所在主机的domain name. 这个domain name在这里刚好也是"corporate.thoughtworks.com",但是它不一定跟puppet master上的domain name就是相同的.然后记下agent的ip地址10.29.2.5. 最后返回到puppet master所在主机上,修改/etc/hosts文件,加入如下映射:

也就是10.29.2.5 clark-laptop.corporate.thoughtworks.com clark-laptop. 由于上面的配置不正确而引出的问题, 在google下还真是每个人都有各自的说法, 试了不少方案, 也算是这个搞清楚了这个流程是怎么回事,希望这个能帮助到刚开始玩puppet的朋友.这样一来ssl握手也解决了, 你可以尝试写一两个catalog让agent请求下来玩玩, 比如安装openssh-server, git , mysql数据库. 另外其实puppet都有一些现有的公共的module,可以避免重新发明轮子, 可以参考http://forge.puppetlabs.com/

Have fun~

接下来应该会尝试下chef, 因为它的语言是使用ruby的, 相对puppet使用自己的DSL, 上手速度应该蛮快的.另外如何使用Chef和Vagrant来测试,应该很有意思.

jit graph Forcedirected add label on edge

2011-06-21T00:00:00+00:00

jit graph(JavaScript InfoVis Toolkit ) is an amazing javascript graph library. Here is the demo i use , the forcedirected graph. Well I just get stuck on how to add the label on the edges. I 've googled for a lot , but lots of doc are out of sync. So i just tweeted a little bit, and found the following solution.

1.modify the json data ,to add a relatedOn key/value

    var json = [
        {
            "adjacencies": [
                {
                    "nodeTo": "sky",
                    "nodeFrom": "earth",
                    "data": {
                        "$color":"#dd99dd",
                        "relatedOn": "from sky to earth"
                    }
                },
                {
                    "nodeTo": "ocean",
                    "nodeFrom": "earth",
                    "data": {
                        "relatedOn": "from ocean to earth"
                    }
                }],
            "id": "earth",
            "name": "earth"
        }
  ]

2.implement a customized edge type

    // init ForceDirected
    var fd = new $jit.ForceDirected({
        Edge: {
            overridable: true,
            color: '#23A4FF',
            type: 'label-arrow-line'
        },

3.specify the edge type to customized type in ForceDirected controller Edge configuration.

    var fd = new $jit.ForceDirected({
        Edge: {
            overridable: true,
            color: '#23A4FF',
            type: 'label-arrow-line'
        },
       Label: {...}
    });

There you go. Following is the result:

Rspec hooks 配置

2011-06-19T00:00:00+00:00

最近项目中有碰到这样一个场景, 我们需要给测试中给每个测试案例置于事务之中,确保方法运行完后回滚所有的数据操作.于是最简单的,在spec_helper.rb中添加如下代码:

  config.around(:each) do |example|
    WAREHOUSE_DB.transaction do
      example.run
      raise Sequel::Error::Rollback
    end
  end
 end

我们这里用的是Sequel来从数据库获取数据, 需要说明的是我们做得项目不是一个独立的,而是嵌在一个大项目中,我们工作的目录主要的是lib/warehouse/下, 而且也只有我们需要使用Sequel. 这样看起来目的是达到了, 后来项目在尝试加速测试的时间, 发现这个时间很多开销在打开事务和关闭事务,而其他项目根本不会用到sequel. 这样一来,也就是说,我们只想让这个事务开启回滚只发生在lib/warehouse/models目录中,而其他目录根本不需要. 那Rspec 有没有这样的功能能配置这个example(测试案例)的条件了? 有.

 #wrap transaction only to model layer of lib/warehouse
config.around(:each, :example_group => { :file_path => /\bspec\/lib\/warehouse\/models/ }) do |example|
    WAREHOUSE_DB.transaction do
       begin
         example.run
       ensure
        Sequel::Error::Rollback
       end
   end
end

其中关键的就是":example_group => { :file_path => /\bspec\/lib\/warehouse\/models/ }" , 这句话指定这个测试案例运行的条件,那就是这个spec文件的路径必须跟"/\bspec\/lib\/warehouse\/models/" 匹配上,才会开启和关闭事务,否则直接运行. 这个其实还满好用的,那现在我们只是用了:file_path这个过滤条件, 哪有没有更加强大的支持了? 有. 参考: RSpec 2.0 and before/after hooks

chronic自然语言来表达时间

2011-05-07T00:00:00+00:00

最近的Rails项目的cucumber测试中，经常会碰到需要处理时间。有意思的是，我们在cucumber测试中需要尽量用自然语言来表达时间，而在其对应的step definition中需要解析成为对应的时间。比如我们需要在feature表达“3个月以前”这个概念，原有的做法是直接在feature中写成"3.months.ago" ,而对应的步定义中则是用到eval()来解析成为对应的时间：eval(time_literal || "1.month.ago"). 也许这个"3.months.ago"还不是非常接近自然语言，为什么中间有"."了？所以我们可以进一步改进一下，在feature中写成"3 months ago" ,然后在调用eval函数之前，先将空格替换成点。嗯，还不错，至少Rails自带支持的时间处理是够用的。但是后来我们需要处理“上个星期的周二”，那这个Rails自带的时间支持无法很好的完成任务了。

所以也就出现了chronic这个gem.

Chronic is a natural language date/time parser written in pure Ruby. See below for the wide variety of formats Chronic will parse.

Chronic能非常的适用于上述的情景。下面摘抄了chronic上的一些例子：

  
  3 years ago
  5 months before now
  7 hours ago
  7 days from now
  1 week hence
  in 3 hours
  1 year ago tomorrow
  3 months ago saturday at 5:00 pm
  7 hours before tomorrow at noon
  3rd wednesday in november
  3rd month next year
  3rd thursday this september
  4th day last week

我们可以直接在feature中写“last week tuesday" ，然后在步定义中直接调用 Chronic.parse(time_literal)就可以很方便的得到我们需要的时间。参考： Chronic 0.3.0 Released: Improved Natural Language Date/Time Parsing

更新: 2010/06/19 Chronic之前用的是2.0.3版本,存在一个问题, 比如说今天是2011/05/30, 使用Chronic.parse("3 months ago") 会得到2011/03/02( 简单的就是减去90天), 而不是像Rails内制的 3.months.ago得到2011/02/28. 我有在github isssue发过一个贴子Parse n months ago problem,得到作者的回复是,在Chronic 0.4.0版本以后都统一使用了ruby自制的时间处理. Ruby 5也贴出了关于chronic更新的消息 chronic update

Mongodb

2011-05-02T00:00:00+00:00

之前数据存储方面接触最多还是Mysql,SQLserver, 用起来还不错, 也没有觉得很复杂, 或许人的思维习惯了复杂,后来慢慢听到了很多很热的词:NoSQL.开始觉得NoSQL == "No SQL", 是表示我们不需要SQL了,放弃关系数据库了吗? 当时觉得这会不会只是一个很热的词了,过阵子就不了了之,就木有花很多时间在这上面.这次有这样一个机会可以好好看看. 很基本的疑问是:关系数据库是70年代产生的,问题是在觉得如何存储和获取数据(通过SQL这样的一个DSL), 而面向对象是在80年代兴起的,也许关系数据库使用了很多的场景,但是到了现在随着WEB大潮的到来,数据量的爆咋,关系数据库已经很吃力了.你实际上在要求70年代设计关系数据库的作者超前设计这样一个"one size fits all"的数据存储系统,很明显这是不太可能的. 所以Not only SQL的数据存储方式就应运而生了. 主要包括4种:

Key-Value store , 比如Memcached
Big-Table , Cassandra
Document-Oriented, MongoDB, CouchDB
Graph Database , Neo4j

本篇文章只简要的介绍MongoDB.

MongoDB (from "humongous") is a scalable, high-performance, open source, document-oriented database.

MongoDB是面向文档的数据库, 这个可以跟关系数据库对比起来.关系数据库我们可以认为是面向表格的,每一个表格都第一行是schema,其他的是对应的数据.一个post-user的例子: 在关系数据库中我们这么表示: 而在mongoDB我们一个文档来表示: 看起来非常熟悉, 对,没错,key-value pair, 非常像JSON的格式.而实际上这个文档的存储方式就是以BSON来实现的.

MongoDB uses BSON as the data storage and network transfer format for "documents".

Schema设计: MongoDB的哲学是:如果它是一个Aggregation Root的model,那么它应该值得有自己的Collection(这里对应关系数据库中的table), 将其它的model尽量内嵌到这个aggregation root中去. 传统的关系数据库很重要的一个概念是范式.我们强调在将Model映射到数据库设计时使用E-R图帮助设计,比如在上面的图中,我们传统是希望将post-user之间联系在设计数据库时划分为post表和user表,它们之间使用外键来创建这样的一个1对1的关系. 而在MongoDB中更强调去范式,尽量能内嵌的就内嵌,而不是去单独抽出来独立成为一个collection.内嵌的方式更加有效率. MongoDB经常要问得是:

"why would I not want to embed this object?"

更多的可以参考MongoDB的Schema Design. MongoDB的Query也是很方便的: 而比较复杂是Aggregation的操作,比如SQL中的group, avg,sum等等,这些操作在MongoDB中可以使用Map/Reduce来完成. MongoDB差不多对主流的编程语言都有很多的支持,在ruby中就有MongoMapper和Mongoid两个很不错的封装框架. Use Case: 我觉得很好的一个例子是日志.现在的方式是将debug的信息都写到文本文件中去.问题就是每当出现一个问题时我需要去那个日志文件中去搜查出错的地方,就非常痛苦.而这个时候使用MongoDB就是一个很好的候选人,它提供很强的查找查询功能,支持快速的读写. 更多参考: http://www.mongodb.org/

cucumber 测试复杂UI

2011-03-14T00:00:00+00:00

最近碰到的一个问题是:我们在controller层都有完备的单元测试,当然既然是单元测试,也就缺乏集成测试了.为什么会是这样了?因为项目要求测试覆盖率为100%,任何在controller层测试中写跟数据库打交道的测试都会通不过项目的质量检查.那既然controller层测试只有一些stub和mock测试的话,集成测试写在哪里了? 在cucumber测试中.那就写呗,难题在于哪了. 就在view层需要渲染一个复杂的报表,使用highcharts库来渲染的图表.通常一般的cucumber测试,在一个scenario中,准备一些数据,然后访问那个页面,查看是否元素或者数据存在.问题就在最后一步,怎么去页面中抓取到想要的元素或者数据,如果只是简单视图显示,可能就很简单,但是如果这个视图渲染出来之后非常复杂情况就难办了. 这就需要进一步分析报表的渲染模型了. highcharts是一个javascript库,你所要做的便是传入合适的数据(配置),然后渲染的事情就可以交给它了. 以这个demo这个基本线条模式为例:

var chart;
$(document).ready(function() {
chart = new Highcharts.Chart({
chart: {
renderTo: 'container',
defaultSeriesType: 'line',
marginRight: 130,
marginBottom: 25
}
..........
//个性化配置的数据
series: [{
name: 'Tokyo',
data: [7.0, 6.9, 9.5, 14.5, 18.2, 21.5, 25.2, 26.5, 23.3, 18.3, 13.9, 9.6]
}, {
name: 'New York',
data: [-0.2, 0.8, 5.7, 11.3, 17.0, 22.0, 24.8, 24.1, 20.1, 14.1, 8.6, 2.5]
}, {
name: 'Berlin',
data: [-0.9, 0.6, 3.5, 8.4, 13.5, 17.0, 18.6, 17.9, 14.3, 9.0, 3.9, 1.0]
}, {
name: 'London',
data: [3.9, 4.2, 5.7, 8.5, 11.9, 15.2, 17.0, 16.6, 14.2, 10.3, 6.6, 4.8]
}]
});

});

所以结合整个流程就是: databaset --> services --> controller -->取道合适的数据 --> view--> highcharts render --> 报表相比一般普通的cucumber测试,现在的问题是highcharts渲染出来后的报表标签(是一个svg)实在是非常混乱,数据被打散了,不像表格一样有明确清晰的意义一目了然. 怎么办了? 问题在哪? 测试对象. 为什么需要测试highcharts它的渲染过程? 那就可以相信highcharts库,不去测试? 问题在于什么是可以自动化测试什么是必须得人工测试. 自动化测试的过程可以在"databaset --> services --> controller -->取道合适的数据" 发生但是关于highcharts渲染的过程,不能包含在自动化测试, 只能由人工测试来覆盖. 这个分解难处在怎么去测试从controller层传回的数据是正确的. 我的做法的一个依据是为什么highcharts渲染过的很难测,因为它数据展现形式缺乏语意性.所以我尝试用一个<table>表格来显示数据.但这个表格只能在测试和QA模式下才可以显示在页面上. 具体做法是:在helper中加入这样一个方法

      def cucumber_debug_print(data)
        return unless ["test", "cucumber"].include?(Rails.env)
        raw_tag = ""              

        raw_tag <<  ""
        transform_on(data).each do |data|
          raw_tag << ""
          raw_tag << ""
        end      
        raw_tag <<" Name
#{data[:name]}"
        raw_tag.html_safe
      end

这样的好处在于:最大限度地将测试包含到自动化测试中.同时提高了一种更有语意的数据表现形式(table),这样测试人员只要根据报表下面的表格中数据来检查报表是否渲染正确,而不需要去scenario中去推导准备数据. 后来证明这个方案是非常重要的,因为后面scenario还涉及到跟其他部分联合的测试,如果采用一般的测试,无法想象这个测试会写成什么样子. 因为之前我跟客户之前的开发人员也有争论这件事情,一开始人家坚持认为cucumber测试丢给QA手工测试就好,结果写出来的测试什么事也没干,因为从controller层传来的数据是应该可以自动化测试的,但是这个时候就必须要QA手工去测试,大大加大的QA的负担,或许像小龙说的这边的开发人员更多的是思考将自己的事弄好就好了,至于QA是否容易测试,我不关心.我一开始无法说服他,人家有多年的经验,挣的面红耳赤也没有必要,那就让事实说话呗. 很高兴的是这个方案更好也更有效的覆盖到了很多单元测试无法测试的问题,给了我们非常即时的反馈. 也许不是一定要非常有侵略性的说服别人,而是让人家自己摸索之后回过头来觉说"哇,你的想法不错阿,试试"

Name
#{data[:name]}

Rails的一些实践

2011-02-14T00:00:00+00:00

这篇文章拖到过年之后才写,真是够拖拉的,厄,不过赶的早不如赶得巧，刚好Ryan Bates更新他的railscastsMetrics Metrics Metrics，而这集跟我要将的东东还蛮有联系的.过年之前我一直在一个公司内部的rails项目试手,因为之前我没有任何的ruby/rails经验,所以这个项目的过程也是有点点坎坷的.我一边在学习知识,同时又将刚刚学到的东西应用出来,就是一个不断学习不断改进的过程.整个学习过程中当然也参考了很多网站拉,比如railscasts阿等等. 那其实整个过程中我都有收获满多的心得,很多之前写的不是elegant的代码,随着后来对其识认识的进一步深入而变的更加的干淨整洁了.我当时也有萌芽个想法去总结这个东西,因为那个时候整个办公室也是有满多跟我一样的rails/ruby新手,如果有这个总结性的东西,也更加有利于知识的分享,更重要的是一个抛砖引玉的效果,让大家的想法经验都能汇集起来.但是一直没有找到这样的一个方式能帮助我更好将心得给表达出来.直到我发现了ihower的rails best pratice这个ppt,这才让我找到了这样一个很不错的方式和参考。

rails best pratice from feedback

View more presentations from clarkht.

实际上了，大部分上面的practice还都是蛮有用的，但是有些地方还真是不一定适用。比如上面提到的使用metaprogramming来减少一些重复，还有一段代码 def self.all_cmts Person.all.select{|p| p.cmt?} end def self.all_hrs Person.all.select{|p| p.hr?} end 以开始我也想得是说使用metaprogramming来减少重复，但是发现写出来的代码实际上是蛮复杂的，考虑到代码的可读性和可维护性，在这个桉例上我们并没有使用这个practice. 那其实还有另外一点值得提到的就是，在session过程进行到permission check这一块时，我其实是没有特别好的注意或办法的，因为这一块我也不是非常熟，所以我想的是把这个问题抛出来，果然徐昊有提出了更好的方桉。相对于我将权限检查这一块扔到模型层（原因是因为需要权限检查将会是在一个非常细粒度的级别上，比如字段，当然我也参考过那个declarative authorization，它应该是能够在字段级别进行检查，或许不是很大问题，但是如何垮字段的检查，不是特别清楚，但是那是一个不同的故事了），而是将权限检查提取到控制层，从而保持模型层的干淨整洁，pretty cool。 Rails Best Practices 投影片 Railscasts Metric Fu Rails Best Practice Gem Rails Best Practice Website 下面是我订阅的關於rails的一些博客. Update: 如何寫出有效率的 Ruby Code by ihower Writing Efficient Ruby Code

关于bundler

2011-01-17T00:00:00+00:00

在rails项目在生产环境中使用的是mysql,而开发和测试环境中使用是sqlite,这种方案使用起来还真是很方便.生产环境是在一台Ubuntu虚拟机上面,而开发和测试环境都是在Mac上面. 在SCM仓库中的Gemfile是这样的:

source 'http://rubygems.org'

gem 'rails', '3.0.3'
 gem 'haml', "3.0.24"
 gem 'ruby-mysql','2.9.3'
 gem 'paperclip', '2.3.6'
 gem 'acts-as-taggable-on', '2.0.6'
 gem 'rdiscount', '1.6.5'

group :test, :development do
 gem 'hirb' , '0.3.5'
 gem 'sqlite3-ruby', '1.3.2'
 gem 'rspec-rails', '2.2.1'
 gem 'mongrel', '1.1.5'
 ....
 end

然后运行bundle install,首先会抛出错误,ruby-mysql gem需要安装Mysql数据库. 然后你需要安装mysql数据库,我使用了很简单的方式 ,详情请看此文Install and configure MySQL 5 with MacPorts

sudo port install mysql5 +server

默认会将其安装到/usr/local/mysql目录下. 然后再次运行bundle install 一切顺利,然后可以干活了. 事情很顺利直到有一天下午下班第二天早上上班时发现生产环境崩溃了,然后就每天都有发生这样的问题.顺便提到生产环境使用的是Passenger+Ngnix作为部署方案. 查看了一下错误日志, 发现里面只有反复的出现一句话

"Broken pipe"

,大家google半天,实在是一是因为大家都是rails新手,没有太多经验;二是这个错误信息太模糊了,搜索了半天也没搞清楚,各种说法的都有, 最后实在是没辙了,就去问米高,米高说这是mysql driver的问题,需要将ruby-mysql gem 换成mysql gem,换过来之后问题解决了.所以这个问题给我的想法是在Ubuntu上安装ruby-mysql时一切顺利,没有出现任何的问题,而在运行过程中却反复出现问题,而且错误信息给的还真是对新手而言不知所云.但一阵沮丧之后,大家要开始编码了,后来发现系统都崩溃了,原来是上次没有将开发过程中也将Gemfile中的ruby-mysql改过来. 那就改过来呗,但却发现安装mysql gem还真是有点麻烦.mysql-gem它因要引用Mysql的lib和headers,使用ruby进行编译和安装mysql驱动.当然你得首先安装了Mysql才行,这样它才行知道Mysql的库(lib)和头部文件(headers)在哪. 安装mysql gem ,如果你在Ubuntu上,

gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

你可能看到很多"Building native extensions. This could take a while... ERROR: While executing gem ... (Gem::Installer::ExtensionBuildError) ERROR: Failed to build gem native extension." 因为在Ubuntu上面,出现这种情况就表示有一些lib或者lib-dev缺失.你可能需要通过sudo apt-get来额外安装这些依赖. 可以尝试:

sudo apt-get install libmysqlclient15-dev

现在运行应该就OK了. 在Mac上面安装其实貌似也不是很困难,至少我感觉比Ubuntu上要简单很多.简单运行

gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

就搞定. 但是我在想能不能有其他的办法了,在生产环境下要求安装mysql gem和Mysql,但是在开发和测试环境中不需要去安装Mysql gem 甚至不用安装Mysql了? 因为在目前这个项目中,我们几乎没有碰到过说要在开发的机子上以production的方式起起来. 那因为Mysql的安装是要在装mysql gem的时候必须要求的,所以能不能在bundler装gem包的时候,在生产环境中装mysql gem而在开发测试环境中装sqlite? 结果是有的.这个答案可以很简单两句话能搞定,但是我想讲讲bundler的工作方式,算是抛砖引入把~~ 一切都很好,就是在Ubuntu上面安装mysql gem貌似有点小小的麻烦,你需要装另外其他的一些本地的依赖.但是它却是又不是非常明确告诉你需要装那些本地依赖,给出的错误信息有点模糊.

关于Bundler

在Gemfile中来描述所需要的一些依赖,然后运行bundle install,下载和安装所需要的依赖,同时将其所安装的gem文件加入$LOAD_PATH路径.同时,你可能也注意到了Gemfile.lock这个文件.那这两个文件的关系就是每次运行bundle install时,bundler会将Gemfile中指定的依赖进行解析,比如在Gemfile中有定义gem 'rails' ,'3.0.0.rc' 那实际上rails 3.0.0.rc还会依赖

rack ~> 1.2.1,而这个依赖的解析和完全展开之后都会写到Gemfile.lock文件中.所以当下次你再次运行bundle install,它会先解析Gemfile的依赖,并检查当前的Gemfile.lock中时候已有这个依赖的解析.如果有,直接使用;如果没有,更改Gemfile.lock,并下载安装对应的gem.

Gemfile语法中还可以指定group,比如上面我的Gemfile,在group :test, :development do *** end 之外的都是属于default这个群.那这里就有三个group,而bundle install会默认执行default group(当然你可以配置), 然后让它选择性的执行某一个group.

source 'http://rubygems.org'

gem 'rails', '3.0.3'

gem 'haml', "3.0.24"
gem 'paperclip', '2.3.6'
gem 'acts-as-taggable-on', '2.0.6'
gem  'roo','1.3.11' 

group :production do
 gem 'mysql','2.8.1'
end

group :test, :development do
 gem 'hirb' , '0.3.5'
 gem 'sqlite3-ruby', '1.3.2'
 gem 'rspec-rails', '2.2.1'
 gem 'mongrel', '1.1.5'
 gem 'cucumber-rails', '0.3.2'
 gem 'cucumber', '0.9.3'
 gem 'spork','0.8.4'
end

你可以看到我现在有将mysql gem单独仍在了:production这个group中的,这个好处是我可以显示的指示我想运行某一个group与否在开发环境中,不希望使用mysql gem ,所以可以调用:

"bundle install --without production"

在产品环境中,不希望使用test/developement中的gem文件,可以使用

"bundle install --without development test"

( 使用bundle install --deployment也许更适合) 其实不就是安装个mysql gem吗? 用bundle install 的不行的原因是因为它需要安装native extention吗? 那能不能在bundler中有一种方式指定了? 答案有两个: 首先,如果你安装的是mysql2 gem的话,它的依赖都会通过它的头部信息而发现,就不需要第二个答案了. 第二个答案是:在bundle config命令尾部加油一个mysql的extention标记:

bundle config build.mysql --with-mysql-config=/usr/local/mysql/bin/mysql_config

以后你运行bundle install都会去使用同一个配置,而这个配置在第一次运行此命令之后保持在#{RAILS_ROOT}/.bundle/config文件记录下来. References: Understanding Bundler Using the New Gem Bundler Today

Using Mootools with Rails

2011-01-12T00:00:00+00:00

最近项目中有需要在Rails中使用mootools，而我们之前使用的是JQuery(ps:是想有什么工作是JQuery办不到了？),这个迁移的过程中遇到了一点问题。过程很简单，网上一搜，找到了与Rails对应的ujs driver , 下载解压后，找到Source目录下的rails.js和Test下的mootools-1.2-core.js，拷贝到项目目录public/javascript/下覆盖掉原来的rails.js(这个是原来JQuery的driver）。最后在application.html.erb文件中的javascript_include_tag中声明上述几个文件。然后。。。。问题出现了，所有按钮无法响应用户的单击操作，同时所有 link_to "My Review", reviews_path 都有解释成POST方法的请求，整个就是一片混乱。思索了良久，还以为是这个框架就是这么设计的，虽然感觉很奇怪，但是也还好，也想出了临时方案。因为你去搜索这个答案还真是蛮恶心的。后来仔仔细细一看，果然有在它在github的issues列表中，提到了这个问题： MooTools 1.3 compatibility 但问题是你不仔细看你还真是不知道，因为它的文档上写的是mootools1.3，但下下来的文件却是1.2的版本，让人有点晕。令人不解的是它却不把1.3放到release中。

rails开发工具的一些tips

2011-01-09T00:00:00+00:00

在这阵子的也不是很长的Rails开发过程中,零星的记录了一些开发过程中使用工具的点滴.

Tip 1:

是对于Rails Console而言,每当我们输入比如Review.all时,它打印出来的总是挤成一团的数据信息. 首先它没有打出SQL语句,而SQL信息有时候对于调试还是蛮重要的. 打开的打印SQL的设置很简单,可以敲rails console之后,输入

ActiveRecord::Base.logger = Logger.new(STDOUT)

紧随其后所有的操作都回打印出查询的SQL语句,很方便. 当然只打印SQL语句也是不够的,为了阅读容易,你可以更希望Review.all不是打印出一团的很难阅读的信息,而是以表格的方式打印出来. 我刚好有看Railscasts有接触到Hirb. 首先gem install hirb一下,然后可以创建一个~/.irbrc的文件中输入

require  'rubygems'
require 'hirb'
require 'active_record'
Hirb.enable
ActiveRecord::Base.logger = Logger.new(STDOUT)

完成两件事,设置logger输出SQL语句,同时打开hirb在窗口中以表格显示. 当然这是全局的设置,如果在某个console中觉得不爽,可以调用Hirb.disable来关闭这个表格输出. 看起来满不错的.(如果你觉得这个屏幕太小,有记录都省略掉了,你当然可以查Hirb的文档,还有办法是进rails console前将你的窗口调大点.这样它的表格的大小也会相应变大.顺便说说iTerm的Command+Enter进入全屏的方式非常爽.)

Tip 2:

每次写完测试,都要跑到窗口中去看rspec spec或者rake spec ,当然你也可以装一个rspec的textmate bundle,从那里面Command+R跑当前文件的测试(ps:我用得最多的还是Rspec Book中作者提到Ctr+Shift+Down来快速地在应用代码和测试代码之间快速切换). 但是每次都有涉及到一些会影响面很大的测试时(这可能会是一个代码坏味),你所想的便是跑rspec spec,但是测试代码一多,这个时间耗费实在太大,一定程度上影响了开发的节奏. 上网搜了搜,发现了这篇文章Kickstart Rspec with Spork,就有谈到使用Spork来加快Rails开发过程中的运行测试的周期.你可以去Spork的github中找到如何配置等等. 如果这其中你碰到有问题,你可以详细的查看它的Issue列表,在这里我先列举出我遇到的一些问题. 我暂时用的是 Spork 0.8.4 + Rails 3.0.3 spork (0.8.4) Authors: Tim Harper, Donald Parish Rubyforge: http://rubyforge.org/projects/spork Homepage: http://github.com/timcharper/spork Installed at: /Users/twer/.rvm/gems/ruby-1.8.7-p302@rails3.0.3 一开始我都是按照Github的配置的来的,基本上没有什么需要个性化的地方.如果你使用过程中发现这个你修改了某个Controller层或者Model层的东西,但是测试仍然失败,而你认为它应该是成功的.那么可能解决的办法是你需要找到environments/test.rb 然后设置cache_classes为false:

config.cache_classes = false

需要注意的是:

It’s important to disable class caching in your test environment otherwise spork will preload all your models and controllers.

关于这个问题,可以查看Issue列表上已经解决的一个问题:gotcha for the docs: rails3 defaults environment/test.rb to preload models via cache_classes setting 而Spork的作者timcharper提到

I've decide that spork 0.9.0 will disable eager loading, regardless of this variable. I may change my mind if enough people are angry about the decision.

但是我暂时使用0.8.4这个版本是没有任何问题的,即它并没有预加载model和controller层的类. (spork -d可以列举出所以已经预加载的类). 现在用起来还真是满爽的,要跑所有测试速度也相对原来快了满多的.爽到BIANG BIANG声之后,遭遇了另外一个问题,同事有碰到一个问题是他在routes里的增加了对应的url到action的映射后,这个测试还是跑不过,提示没有对应的route有定义.我挣扎了半天,突然想到两个星期前原来Xia姐也有问过我,但当时我也没搞清楚,只能关了Spork,重启才行.额,联系到Spork的原理,我想先看看Spork会预加载哪些类了? 运行"spork -d" 一看:

果然有routes.rb被预加载了,所以每一次运行rspec 命令,都不会去重新加载routes.rb这个文件,哪怕你在里面做了惊天地泣鬼神的变化. 所以问题是如何手动的指定了让Spork每次都重载这个文件了? 夷,可以参考Issue列表,说不定有人也有这样的问题.一看,没有找到跟我问题一样的,但是却发现有点类似的,那便是Factory girl with bundler and spork - no reloading of classes. 那我知道我应该去spec_helper.rb中的Spork.each_run 方法(默认是空)中require我门需要的文件config/routes.rb, 所以下面便是修改之后的:

Spork.each_run do
#make sure each time running rspec , it will reload routes.rb file.mod
require File.expand_path("../../config/routes", __FILE__)
end

启动Spork,运行rspec,改动routes.rb文件,立马就能在测试中显示出效果,赞! 所以基本上以后的开发都不会需要改动Spork了,也不会有问题跟它有联系了,因为它预加载就是下面几个文件: config/application.rb config/boot.rb config/environment.rb config/initializers/backtrace_silencers.rb config/initializers/inflections.rb config/initializers/mime_types.rb config/initializers/secret_token.rb config/initializers/session_store.rb spec/spec_helper.rb 一般来说,对于开发,基本上改动的范围无外乎route, model ,view 和 controller等,上面列举的文件不需要去改动它们. 那如果以后你发现了诡异的问题,测试应该过,却要是没有过,那就按照上面的方法来办吧. 最后,在Textmate中Ctrl+Command+Alt+V可以跳出剪贴板的所有历史记录,直接输入前几个字母就能定位,这个很实用. esc可以有一定程度的补全,也很酷看一个这个链接: 10 TextMate bundles/plugins to boost your Ruby on Rails Code Beautifier 格式化代码满好使的,AckMate搜索很方便等等

Update @2011/02/11 : How To Get Rails 3 and RSpec 2 Running Specs Fast (From Scratch)

减少HTTP请求

2010-12-25T00:00:00+00:00
    这篇文章的一个基调是根据Yahoo! Exceptional Performance Best Practices中第一条规则"减少HTTP请求".
为什么需要减少HTTP请求次数?
    当你打开一个网页时,浏览器就会向服务器端发送请求,这些请求包括有图片,CSS,Javascript和Flash等等,然后服务器对这些请求做出响应,将对应的资源发回给客户端.这个过程中用户需要等待这些响应,而服务器需要分配资源来处理请求,对于一些小型的网站,浏览的人少,那么对应的服务器的压力也不会太大,响应速度也不至于太慢(当然这也跟网速有关).但是对于大型的网站,每秒中浏览人的量是非常大的,那么服务器的所承受的压力也会变大,到了一定程度,就会出现处理不过来的情况,对于用户而言,就是看到一张空白的网页,难以提供良好的用户体验.这里可以参考 "Performance Research, Part 1: What the 80/20 Rule Tells Us about Reducing HTTP Requests" by Tenni Theurer on the Yahoo! User Interface Blog.
Table 1 shows popular web sites spending between 5% and 38% of the time downloading the HTML document. The other 62% to 95% of the time is spent making HTTP requests to fetch all the components in that HTML document (i.e. images, scripts, and stylesheets). The impact of having many components in the page is exacerbated by the fact that browsers download only two or four components in parallel per hostname, depending on the HTTP version of the response and the user's browser. Our experience shows that reducing the number of HTTP requests has the biggest impact on reducing response time and is often the easiest performance improvement to make.

Table 1. Time spent loading popular web sites

Time Retrieving HTML Time Elsewhere

Yahoo! 10% 90%

Google 25% 75%

MySpace 9% 91%

MSN 5% 95%

ebay 5% 95%

Amazon 38% 62%

YouTube 9% 91%

CNN 15% 85%

    既然关键提高性能的关键之一是在于减少HTTP请求次数,那么什么东西会引起这些请求了? 常见的便是下面几个方面:CSS文件,Javascript文件和图片.     对于CSS文件,我们可以使用将很多零散的CSS文件合并并且压缩(比如使用Gzip,经过压缩之后大小只有原来文件的70%-80%,相当客观的成绩),然后在传送到浏览器端.在Rails中,注意到在application.html的title标签中
stylesheet_link_tag :all, :cache => true
米高说这个这个all的属性使得Rails在产品环境下将所有位于public/stylesheets目录的css文件进行合并成一个叫all.css文件,可以减少HTTP请求的次数,很有用的一个功能     对于Javascript文件,同样的也可以对其进行压缩,比如常见的JQuery就有压缩之后的版本jquery.min.js.     对于上面二者之前,大概可以归纳出减少HTTP请求次数可以通过合并压缩的办法来达到.对于图片是不是也可以借鉴了?
CSS Sprite
    减少HTTP请求次数中有一条:
CSS Sprites are the preferred method for reducing the number of image requests. Combine your background images into a single image and use the CSS background-image and background-position properties to display the desired image segment.
    CSS sprite的想法很简单,就是将零零碎碎的图片合并为一张图片,并通过background-position来在大图片中定位到自己需要显示的区域.     对于下面实现一个简单的一般的我们习惯将弄十张这样的小图片：
ul li a#doc_link{ background-image:url("../images/doc_grey.png"); } ul li a#doc_link:hover{ background-image:url("../images/doc.png"); } ...
    但是我如果使用CSS Sprite的话,就是简单一张“大”图片:     在这个过程中,我发现合并这些图片对于我这样一个photoshop盲来说,很吃力,所以我很明智请求别人帮忙.而且合并完成之后还需要一张一张图片计算这偏差,更是显示的有点"得不偿失". 所以大牛Steve Souders在这篇SpriteMe makes spriting easy中宣布了SpriteMe,而且他此文中把SpriteMe和另一个制作图片精灵的工具CSS Sprite Generator做了一个比较. 个人感觉SpriteMe非常强大,特别是它有一个选项"Share your savings"能提交你使用Sprite之后数据的变化,包括文件大小,请求的次数浮动等等. 但是有一个问题是当你的页面没有出现"suggestions"时,这个"new sprite"根本不工作,这时只能使用CSS Sprite Generator了.     那这个时候我想到能否将css sprite跟rails集成起来了,一搜果然有人提供了这样的支持.黄志敏在其博客CSS Sprite Best Practice中就有提到他自己写的一个支持.你可以在这里了解到更多关于使用和安装的细节:Use css sprite automatically.经过半个小时的摸索,我发现它的好处在于它提供了更小粒度对图片操控的能力,同时接管了很多繁琐的任务,比如如何合并图片,按照哪种方式来产生对应的CSS等等. 但是也有问题在首先它的安装过程过于纠结,它需要依赖rmagick,但是rmagick装起来有点麻烦(MacPorts is the devil, and shouldn’t be installed just for installing ImageMagick).另外一个是对于png图片的生成出现了透明度丢失的现象,对于JPG图片却是正常的.     ps:Dave Shea在Sprite Optimization中提到了他在拼接png图片时,并不会使用Adobe compressor,而是其他的压缩工具Pngcrush 或者 PngThing ,我不太清楚黄志敏在实现css_sprite插件用得是什么样的压缩工具.     那是不是CSS Sprite没有负面效应了? 你肯能想到了,如果这张Sprite图片太大了,那么浏览器所需要的内存会爆了?     确有,Andy King在CSS Sprites: How Yahoo.com and AOL.com Improve Web Performance提到在iphone上,过大的图片会占用手机本不是很大的内存,可能导致一些问题,并指出:
To maximize accessibility and usability, CSS sprites are best used for icons or decorative effects.
    Firefox工程师Vladimir就在To Or Not To Sprite 文章提到了使用CSS Sprite有可能会显示大图片时肯能产生的内存问题. 这里也顺便引用他关于使用CSS Sprite应该注意什么时的一段话:
There are cases where CSS sprites are clearly beneficial; the main example is combining a bunch of small similarily-sized images into a single one. For example, a bunch of 16×16 icons used as indicators for various things on your site, or a bunch of 32×32 icons used as category headers or similar. But combining images with vastly different dimensions, especially when there are skinny and tall images in the same sprite as wide and short ones is almost never a good idea.

Inline Images
    减少HTTP请求次数中还有有一条: Inline images use the data: URL scheme to embed the image data in the actual page. This can increase the size of your HTML document. Combining inline images into your (cached) stylesheets is a way to reduce HTTP requests and avoid increasing the size of your pages. Inline images are not yet supported across all major browsers.
base64-Encoded Data URIs
    对于这个我的理解是使用一中更高效的encode方式将图片(或者其他东西,比如HTML)的内容字节转换成一种更精短的字节,这些转换后的字节足够小以致于我们可以直接在CSS文件中描述出来,同时定义相应的条约让浏览器在解析的时候能根据这个条约将字节反向还原过来. 要理解这个,得弄清楚URL跟URI的区别,在Nicholas C. Zakas的文章Data URIs explained中有详细说明它的工作原理,强力推荐. 想象一下在一个图片非常多网页中,这个HTTP请求的次数将会减少多少,因为每一张图片都不需要发送任何请求了.这样以来相对CSS Sprite ,Nicholas都甚至喊出来"Data URIs make CSS sprites obsolete" 的口号. 但是你会疑问说,那么你也看到了CSS文件会变得越大阿?不会得不偿失吗?     那这个确实你可以通过Gzip对CSS文件进行压缩,压缩之后大小就跟你原来写图片引用的CSS文件是差不多的,所以这个不是一个问题.但是它的浏览器兼容怎么样了?
Web Browser Support
    Data URIs are supported in:      Firefox 2+      Safari – all versions      Google Chrome – all versions      Opera 7.2+      Internet Explorer 8+     你有可能看到了,它并不支持浏览器(MOSG都还是很不错的),特别是IE8之前的系列.IE是一个怪胎,所以如果你想在IE8之前的ie浏览器中支持URI的话,可以参考MHTML – when you need data: URIs in IE7 and under. 当然Data URI依然面临一些问题,最突出的便是相对于原来图片可以在客户端被缓存,而现在描述图片的内容都在CSS文件中,一旦对于CSS文件有最微小的改变,也会导致整个代码需要重新下载,比较麻烦.     使用:     IE本来就不应该成为你忌讳某个工具的原因,所以及时你不得不这样,Nicolas其实已经设计好了这样一个工具cssembed,它可以让你选择输入的CSS文件,然后遍历文件讲图片引用转码成Base64.你完全不用担心IE的问题,它都有处理好了. 下载好JAR文件后,你可以通过这样的调用
java -jar cssembed-0.3.2.jar styles.css > styles-base64.css
    这样以来就可以将style.css文件中所有的图片引用转码成为base64格式,并输出到style-base64.css文件夹中,就那么简单.
ul li a#css_link{ background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIAAAA..省略...) }
    使用传统方式: Style.css 大小没有压缩:4kb 加上10张小图片的大小:137KB ,总共是141KB,10次请求图片     使用CSS sprite: Style.css大小:4kb 加上Sprite图片大小:85kb ,总共是89KB, 1次请求图片     使用Data-URI Base64没有压缩:194kb 使用YUI Compressor压缩之后貌似仍然是185kb,0次请求图片.         CSS Sprite和Data-URI:
CSS sprites were a solution to the problem of multiple HTTP requests to download multiple images. Data URIs allow you to embed images directly into your CSS files, solving the same problem in a much more elegant and maintainable way.
    最后介绍一下Rails下强大的库Jammit,它不仅支持对CSS和Javascript的像YUI compressor类似的压缩功能,同时它还支持Data-URI / MHTML image and font embedding.
引用
    Dave Shea在A List Apart的博客: CSS Sprites: Image Slicing’s Kiss of Death     在09年的跟进CSS Sprite更新了一篇博客:Sprite Optimization     Chris Coyier在CSS Tricks的非常棒的一篇文章 CSS Sprites: What They Are, Why They’re Cool, and How To Use Them     CSS-Tricks CSS Sprite Screencast: How to Use CSS Sprites     Robert Nyman;How To Reduce The Number Of HTTP Requests     Rob Larsen的文章: Why Front End Performance Matters to Everyone, Not Just the High Traffic Giants     黄志敏(flyerhzm)关于CSS sprite best practice 的PPT: Css sprite best practices     提高性能另外一个关键在于异步加载提高并发,绕过传统的阻塞模型,实现网页的迅速加载.以后会继续提到"non-blocking script",有意者请看Steve Souders的ControlJS系列和Nicolas的Thoughts on script loaders.         嗨,这一天我的Reader可是多了不少的订阅阿~~ Update:关于更多DataURI的内容可以参考:Wiki的data URI scheme

Follow Your Hunch

2010-12-17T00:00:00+00:00
    这篇博客的起由是前阵子在了解GWT(Google Web Toolkit)的时候,因为GWT内置了一个Java到Javascript的编译器,所以你可以在Java平台上进行开发,好处在于你并需要有对Javascript,跨浏览器等等很深的认识,甚至你可以是个文盲.当然这种好处的代价在于你以Java语言来开发客户端,这个过程极其貌似原来开发Swing的过程.所以下面是一段很常见的代码,给一个按钮添加一个点击事件:
final Button closeButton = new Button("Close"); // Add a click handler to close button closeButton.addClickHandler(new ClickHandler() { public void onClick(ClickEvent event) { dialogBox.hide(); sendButton.setEnabled(true); sendButton.setFocus(true); } });
    看起来很简单的过程,当用户单击"Close"按钮时,事件就会出发,在处理体中就会隐藏掉对话框,发送按钮聚焦等,看起来就是我们写SWING的风格阿,多熟悉. 那但是当我跟着GWT官网的一些例子走时,我愈来愈发现当项目稍微大点(哪怕这还只是一个演示项目),这种添加事件的场景随处可见,漫天飞舞. 哪怕有时候我的事件处理简单到只是显示一个对话框,
someButton().addClickHandler(new ClickHandler() { public void onClick(ClickEvent event) { //do sth } });

Why Bother?
    我依然还得将这个模板写一遍,非常单调无聊.写多了,除了麻木,就会想想了,感觉这个太不直观了,简直就是要让你得大脑大圈,有没有更好的主意了?     鉴于自己经验不是很多,然后就问了问了JP,在Ruby中怎么弄的.讨论着讨论着,突然灵光一闪,想到一个极佳的案例,这个案例来自OSGi in Practice这本书, 简要的描述一下这个问题: 现代Java GUI编程大部分是单线程的,比如SWING和SWT,那么每当我们更新界面的某一个组件,而这个组件或者库是来自于GUI的库,比如更新一个label的信息: label.setText(newMsg) ,那么我们必须确保这个调用是发生在GUI的那个线程上面,也就是默认的"Event dispatch thread"之上. 所以如果这个调用在其他线程上,解决方安就是将这段代码传给GUI库支持的一个工具类SwingUtilities.在Swing中,需要做得就是将这段代码扔到一个Runnable的实例中的run方法体中,调用SwingUtitlies.invokeLater方法来执行.当然,我们可以先通过EventQueue.isDispatchThread来确认一下是否我们现在就是在GUI这个单线程上,如果在就不需要SwingUtilties,直接调用就好.     那如何在Java中实现对这一段模板的封装了(可以想象在SWING中有地方会需要这个处理)? 联想到刚才的按钮的一些单击事件的处理,你可以想到使用实现了Runnable接口的"匿名类"
private void updateDisplay(Runnable action){ if(EventQueue.isDispatchThread){ action.run(); }else{ SwingUtitlies.invokeLater(action); } }
那么以后我可以讲更新标签的动作就可以这么做了:
updateDisplay(new Runnable(){ public void run(){ label.setText("this way is just too CUMBERSOME!!!!!!!!!!!!!!!"); } });
    看起来还不错,那么以后其他的更新界面的东东都可以这么做. 但是写了几次了,还是感觉这个有点太笨拙了,就一行代码的动作都得写一些什么new Runnable ..... 看起来好分心阿. 那能不能这样了?
private void updateDisplay(Expression expre){ if(EventQueue.isDispatchThread){ expr }else{ SwingUtitlies.invokeLater(new Runnable(){ public void run(){ expre; } ); } }

updateDisplay(lable.setText("this way is just what we wanted!"));
    那这个在Java中可以实现吗? 答案是在Java语言上是不行的,你无法阻止这条expression被立即执行,所以在进入updateDisplay方法体之前,这个expression也就是更新标签的调用是会立即被触发的.那使用匿名函数实际上让这个方法的调用者承担了更多责任,你得负责实例化这个匿名函数,然后讲你真正想执行的代码片段仍到里面,非常不可取.但问题是Java语言它没有将这个复杂性囊括到自己本身的库中,结果就是我们得写很多的匿名函数的Boilerplate code. 那你可以说我可以写个什么插件让IDE像自动产生getter和setters一样产生匿名类的插件不久可以了? 这个确实是一个很好的workaround,同时这个行为很符合insanity的定义,
Insanity: doing the same thing over and over again and expecting different results.
真正的问题还是Java语言不够强大,为什么我需要这么做,为什么将这些笨拙转嫁给了我们? 拜托,好歹你也遮遮丑嘛!         关键:     其实问题的关键在于我们能否在推迟方法参数中表达式的evaluation,让我们能操控表达式在何时被解析,而不是像Java语言中你没法做到,因为表达式在进入方法体之前就被解析了.     所以下面将介绍使用Scala和Clojure两种现在赤手可热的语言来实现我们所想要的方式.
Scala Way
    首先我们来定义updateDisplay这个方法:
def updateDisplay(action: => passedInAction) = if(EventQueue.isDispatchThread){ action }else{ SwingUtitlies.invokeLater(new Runnable(){def run = action}); }
那现在你可以很随意的使用:
updateDisplay(lable.setText("this way is the AWESOME Scala way!"));
简单到一行搞定,可读性也提高了,更重要的是你看到调用者都需要关心什么匿名函数.     Scala是怎么做到作为方法参数中表达式的这样一个延迟解析或者执行了?     关键在于updateDisplay定义方法参数的片段,我们使用action : => passedInAction .那在Scala中,前一个action实际上定义了别名,它指向了后面传入方法体的真正参数. 为什么不是action : passedInAction?因为在Scala中这种方式表达的效果跟Java语言的解析是一样的,passedInAction会在进入方法体之前被执行,在Scala中称为"Call-by-value".而action: => passedInAction 相对前者增加了一个' => ' 符号,魔力在于,它使得passInAction不会被立即解析,而是当你在方法体中真正调用action这个别名时,它才会被触发执行,这种方式在Scala中称为"Call-by-name".     这样在看起来代码就非常简洁明了,非常棒!
Clojure Way
    相对于Scala,Clojure作为一门Lisp的方言,自然而然拥有Macro系统.尽管Scala没有macro的概念,但是它拥有很好非好强大的facilitators,就像前面看到的"cally-by-name",来避免使用Macro.Macro有些时候就是怪兽,(当然我不知道是不是Scala的作者也正有此意),因此怎么去写好Macro就需要一定的规则或指导Rules for writing macro. 使用Macro有几个要注意的地方,我选取其中一个
you can write any macro that makes life easier for your callers when compared with an equivalent function
在Clojure中,下面便是Macro的实现
(defmacro update_display [expr] `(let [is_on_dispatch_thread? (EventQueue/isDispatchThread)] if is_on_dispatch_thread? ~expr (. SwingUtitlies invokelate (proxy [Runnable] [] (run [] ~expr)))))
调用:
(. update_display (. label setText "this is the BRILLIANT Clojure way!"))
    乍看起来还不习惯了,那么多括号来括号去的,这些括号便是Lisp的风格.读起来的话,如果你熟悉LISP的风格,那么读起来也还是蛮直白的,let用语绑定前面一个别名到后面的那个值,我们首先调用(EventQueue/isDispatchThread),实际上因为Clojure跟Java这样一个interoper*,它实际是调用EventQueue.isDispatchThread()其中/表示静态方法调用.如果is_on_dispatch_thread?为真,也就是在GUI的线程上,那么直接调用expr,也就是(. label setText "this is the BRILLIANT Clojure way!"), 如果不为真,则将此段代码封装到一个实现Runnable的接口的类中,也就是proxy关键字的作用.(ps:我现在对Clojure也不是非常熟,可能有更简洁的写法.)     同样的,Clojure实现的方式也是很简直明了的.
Follow Your Hunch
    我记得原来郑晔有讲过他学习新语言的时候的一个感受,就是Scala里发现根本就需要有Visitor模式的存在.为什么了?因为它强大的平台已经讲这些额外的复杂性都考虑进去了,让你--作为一个调用者Caller更加专注于你的领域,简化你的使用.谈到这个,在Java中如果你发现一些重复出现的模式,然后你想去封装这些模式,你所能操控的最小粒度只能在方法级别上.比如,if 是Java中一个关键字,它定义有自己的语法,当你发现有关于if使用的重复出现的模式时,你是没有任何办法去封装这个模式,因为if 它不是一个方法,更不是一个数据.你不能创建一个unless 关键字,相反你只能通过 instead of using     unless (tickExpired()) 你所能做得便是:     if(!tickExpired()) 或者写一个相反的方法来提高刻可读性     if(tickNotExpired()) 的方式来绕过这个问题. 也就是你不可能给Java语言添加新的功能,而这正是Clojure中Macro可以做到的,因为一切的关键字都是Clojure数据 "Code as data".迫使你的大脑做从if(!foo)到unless(foo)的转换,看起来是一个很微小的问题,但是它隐藏的真正的问题是:
Every distinct syntactic form in the language inhibits your ability to encapsulate recurring patterns involving that form.
我想引用Stuart Halloway在Programming Clojure 中的关于macro的一段总结:
The Wikipedia defines a design pattern to be a “general reusable solution to a commonly occurring problem in software design.” It goes on to state that a “design pattern is not a finished design that can be transformed directly (emphasis added) into code.”
他继而说道:
That is where macros fit in. Macros provide a layer of indirection so that you can automate the common parts of any recurring pattern. Macros and code-as-data work together, enabling you to reprogram your language on the fly to encapsulate patterns.
这个也使我想起来米高原来做过一个讲座有提到"跟着直觉"走,如果你感觉很不爽,你应该问问有是不是什么地方不对,有没有提高的地方.同时也使我想起了Twitter的创始人Evan Williams在TED大会坐的一个讲座Listening to Twitter users中提到的"Follow your hunch".
跟着直觉走
    PS: 我本人对于Scala并不是很了解,对于Clojure的了解程度尽在于读了读<Programming Clojure>,所以这里如果有解释不是非常清楚的地方,请各位帮忙指出来.
引用
关于Scala:     郑晔翻译的 : Scala程序设计：Java虚拟机多核编程实战 Scala 'Call-by-name' and 'Call-by-value'     What's the difference between a lazy argument, a no-arg function argument, and a lazy value? 关于Clojure:     Programming Clojure, by: Stuart Halloway Rich Hockey 的讲座     Clojure for Java programmer Uncle Bob关于为什么选择Clojure的文章:      The Clean Coder: Why Clojure?

将UI和领域模型隔离开来 (二)

2010-11-30T00:00:00+00:00

目的:
那我们现在要做得便是要借助Hibernate Validator的支持,将整个判断逻辑从客户端移植到服务端.
解决方案:
关于这个问题,我记得当时有跟Kai讨论过,提出了两种方案. 第一种,Kai的方案,扩展Spring MVC的WebDataBinding功能,但是我们发现这样我们能操控的行为,只能在发生数据绑定这个过程之后. 也就是说如果我想在数据绑定之前,将请求中的month和year字段合并为一个字段storyDate,这样Story这个对象仍然能保持与UI的隔离.这就要求我们能控制数据绑定之前的行为,Kai说他先去SPIKE下,我现在接着把这个STORY继续下去,因为时间很紧张,如果完成了这个之后有时间就采用Kai的办法. 等他搞完,我跑过去一看,他说搜遍了文档,也没发现Spring MVC提供这个钩,所以这个Story里面还是得有month和year字段,只是说扩展了Spring的Validator,实际上就是一个Proxy模式.请看代码:
@InitBinder protected void initBinder(WebDataBinder binder) { binder.setValidator(new StoryPostBindingValidatorProxy(binder.getValidator())); //这个动作发生在绑定之后,迫使我们依然得保留month和year字段 }
在StoryPostBindingValidatorProxy中是这样的:
@Override public void validate(Object o, Errors errors) { if (o instanceof Story) { Story story = (Story) o; if(story.getYear().isEmpty()){ Calendar calendar = Calendar.getInstance(); story.setStoryDate(calendar.getTime()); errors.rejectValue("storyDate" ,null ,"Year cannot be empty."); }else if(doesYearContainsNumbers(story.getYear())){ Calendar calendar = Calendar.getInstance(); story.setStoryDate(calendar.getTime()); errors.rejectValue("storyDate" ,null ,"Year should only contain numbers."); }else{ DateTime date = story.getMonth().equals("-1") ? new DateTime(Integer.parseInt(story.getYear()), DateTimeConstants.JANUARY, 1, 0,0,0,0) : new DateTime(Integer.parseInt(story.getYear()), Integer.parseInt(story.getMonth()), 2, 0,0,0,0); story.setStoryDate(date.toDate()); } } springValidator.validate(o, errors); } private boolean doesYearContainsNumbers(String year) { return !(year.trim().matches("[0-9]{4}")); }
可以看到在进入springValidator.validate(o, errors);之前,我们有检查year和month字段,填入合适具体的错误信息,同时将storyDate设置一个正确的时间.这样的话,好处在将如何组合成正确storyDate的逻辑提取了出来,减少了Story模型的污染,同时也提高了代码的可测性. 在Story中StoryDate的描述:
@Column(name = "story_date", nullable = false) @NotNull @Past(message = "Did you invent time travel? Please enter a date in the past.") @NotBeforeMay1954(message = "Date cannot be earlier than May,1954.") public Date getStoryDate() { return storyDate; }
前三个annotation都是Hibernate内嵌支持的,后面一个是我自己实现的,所代表的意思非常明了,StoryDate不能早于1954年5月,这样以来的好处在于StoryDate在它上面只需关心它自己的逻辑,跟UI都无关,同时代码的可读性也大大增强. 下面是NotBeforeMay1954的实现图,详细的可以参考Hibernate Validation Annotation中的Customization 面临的一个实际问题是: 当用户浏览某一个故事,并单击Edit时,此时进入edit_story页面,它需要显示故事的内容,标题,年和月等,来等待用户编辑.
@Transient public String getMonthOfStoryDate() { if (storyDate == null) return ""; if (new DateTime(storyDate).getDayOfMonth() == 31) { return "0"; } else return String.valueOf(new DateTime(storyDate).getMonthOfYear()); } @Transient public String getMonth() { return month; }
那如果你页面中绑定到month这个字段,此时页面就无法显示真实的月份,因为month被标记为@Transient; 如果绑定到monthOfStoryDate字段的,当你编辑完成准备提交时,页面上month字段的值却要无法绑定到Story模型上的monthOfStoryDate字段上,因为Story只有month这个字段. 为了解决这个问题,最终的结果是: 在StoryController中,
@RequestMapping(value = "story/{storyId}/edit", method = RequestMethod.GET) public ModelAndView edit(@PathVariable int storyId, @CurrentUser User currentUser) { Story story = storyService.findById(storyId); //..... modelAndView.setViewName("edit_story"); story.setMonth(story.getMonthOfStoryDate()); modelAndView.addObject("story", story); //.... }
这看起来很tricky阿! 仔细思考,为什么会有这个问题了? 原因还是在于Story作为一个领域模型,它有自己的biz rules,它本应该不受到UI层的影响或者耦合的,但是在这里Story却是跟UI紧密相结合. 你可以想象,写代码者当时拿到这个故事时,先弄了页面,发现需要year和month两个输入,于是自然而然在Story中也添加了对应的两个字段,也许当时他却是很方便,但是却给现在尤其是维护或者扩展时带来极大的麻烦. 为什么领域模型模型需要关心你的UI 了? 为什么不从模型本身所要承载的biz rules出发了思考问题了? 为什么要倒过来先写UI然后说看看模型层应该相应的怎么变化,这个不应该是模型层相对稳定且独立,而UI层比较趋于变化不? 本末倒置. 在这里思考的是从Story出发,我需要的就是一个StoryDate,我不关心你是怎么是构成这个日期的,而且我也不能去关心那个,不然别人怎么能够放心大胆的重用我了? 举个例子,可以方便你看出Story模型跟UI有多么高的耦合,如果将来页面UI需要引入day,那岂不是我要得改Story模型,可能我需要引入day这个字段,同时想一想那个它怎么判断用户是否输入月份的逻辑,在想想它在页面如何判断取出的Story是否有month需要显示出来(那个"-1","0" 和"31"), 不可思议的恐怖,真得是牵一发动全身,这个rippling effect实在是无法承受滴,试问你如何去维护这样的代码. 从OO的角度来说,它的一个原则似乎也可以应用到这里:DIP. 所以也可以看到即使你应用到了MVP模式,也并不一定代表你能很好地掌握它的应用场景和上下文.
Polishing the design
我提出的第二个方案是,为什么要在数据绑定的时候,直接使用模型来作为绑定的载体了? 如果说UI 很简单,能与模型一一对应,比如在这里如果没有year和month,那用直接使用模型作为数据绑定的载体很OK,如果将来UI变更了,我再做以下的事情就好:创建一个于表格对应的ViewModel(或者说Presenter). 这样做的想法很简单,每一个页面都代表这当前模型或者系统的一个状态,而这些状态不一定跟真实的模型能一一匹配上,正如表格中month和year字段,此时在Story模型并没有相应的字段与之对应.这个问题就非常像ORM中的Impedance Mismatch,也就是说正如面向对象和平面的关系数据库之间有鸿沟,因而你有看到了什么RowMapper之类的,而在这里它正相当于Presenter或ViewModel,它来负责信息的一些转换. 这个问题使我想到我原来做过一个关于PRG模式的讲座,当时准备材料,我看了一篇文章 Redirect After Post,其中有一段,当时我不是很懂,但现在想想貌似还蛮有道理的. 所以了,我门将会在Story模型中去掉month和year字段,我只关心给我一个日期格式的storyDate,我拿过来用就好,我不care它怎么得来滴,所以这个使我想到了"Ask don't tell"的原则,在这里真是很好的fit in阿.除此之外所以的判断逻辑都将从Story模型中移除出去,去哪了? 正是NewStoryViewModel. 这里我们将创建一个NewStoryViewModel对象,并使用它作为数据绑定的载体,那么它肯定会包含
public class NewStoryViewModel { private String title; private String content; private String city; private String month; private String year; public Story toStory(){ Story story = new Story(); story.setTitle(title); story.setContent(content); story.setCity(city); story.setStoryDate(getRightStoryDate()); return story; } private Date getRightStoryDate() { return getMonth().equals("-1") ? new DateTime(Integer.parseInt(getYear()), DateTimeConstants.JANUARY, 1, 0, 0, 0, 0).toDate() : new DateTime(Integer.parseInt(getYear()), Integer.parseInt(getMonth()), 2, 0, 0, 0, 0).toDate(); } //...getters and setts
在StoryPostValidationProxy中:
@Override public void validate(Object o, Errors errors) { NewStoryViewModel newStoryViewModel = (NewStoryViewModel) o; if (newStoryViewModel.isYearFieldEmpty()) { createRightStoryForValidation(newStoryViewModel); errors.rejectValue("storyDate", null, "Year cannot be empty."); } else if (newStoryViewModel.doesYearContainsNumbers()) { createRightStoryForValidation(newStoryViewModel); errors.rejectValue("storyDate", null, "Year should only contain numbers."); } springValidator.validate(newStoryViewModel.toStory(), errors); } public void createRightStoryForValidation(NewStoryViewModel newStoryViewModel) { Calendar calendar = Calendar.getInstance(); newStoryViewModel.setYear(calendar.get(Calendar.YEAR)); newStoryViewModel.setMonth(calendar.get(Calendar.MONTH)); }
这样的话,我们就得到了一个非常清晰明了的设计和实现.现在的Story模型,完全不受到UI的影响,这些原来的耦合都被封装到了NewStoryViewModel之中,同时也大大提高了Story模型的可重用性和NewStoryViewModel以及验证逻辑的可测性.那你现在还可以进一步将NewStoryViewModel类中方法getRightStoryDate()做进一步的改进,你可以封装这些有意义的字段,你甚至可以将其移动到一个类似Constants的类中,这将极大方便你日后显示或者其他的. 比如在显示故事的时候,原来的代码是在StoryViewModel中
public EscapedString getDateOccurred() { if (story.getMonth().equals("0")) return escape(story.getYear()); else return escape(new DateTime(story.getStoryDate()).monthOfYear().getAsShortText() + " " + story.getYear()); }
你可以该进为:
public EscapedString getDateOccurred() { if (StoryViewModelUtil.isYearOnly(story.getStoryDate()) return escape(story.getYear()); else return escape(new DateTime(story.getStoryDate()).monthOfYear().getAsShortText() + " " + story.getYear()); }
等等有很多可以提高的地方. 总结: 很多问题有时候不是浅层次的问题,而有可能是设计问题. 如果当初写此代码之人写完之后仔细一看,(当然拉,感觉也不会,因为一个测试也没有,代码都成这样了),感觉这个看起来很不爽,继而能通过询问他人或者其他方式来思考这个问题,恐怕也不会造成现在我门维护代码如此之困难的窘境.所以Bob的Craftsmanship宣言很有道理.

Not only working software,

but also well-crafted software

将UI和领域模型隔离开来 (一)

2010-11-30T00:00:00+00:00
有将近一个月没有更新博客了,这阵子确实比较忙,必须淡定,克己轩. 不过这期间我有在Evernote中记录了一些零星的想法和感触,今天刚好有时间,于是拿出来仔细瞧了瞧.其中有一篇是关于我在TWU即将结束的时候记下来的日记,记得当时有一个故事,需要将浏览器端的javascript验证逻辑移植到服务器端.在完成这个故事的过程中给我留下了很深的印象,遇到了很多有意思的问题,我觉得我有必要写出来,否则时间过久了就会忘了.TWU这个项目比较有意思,它历经了几届TWU学员的蹂躏,很好的完成了试验品这个角色赋予的任务,你可以猜到当我们拿到这个项目时,并不是推到原有的而从新开始开发,而是在既有的代码库上进行开发. 我会选取我在完成这个故事中的一个情景入手,这个情景是用户可以跳到"创建新故事"页面,然后可以输入title(文本框,必须填写),city(文本框,必须填写),content(文本框,必须填写),month(下拉框,非必须选择)和year(文本框,必须填写)这几个选项,然后单击"Publish"按钮进行发布,发布之后会跳到显示故事的页面,它会显示刚刚创建的故事的一些信息. 其中值得注意的是用户在"创建新故事" 页面的表格中,如果用户跳过month字段,因为它是非必须的,那么在显示故事的页面的"Event Date: "后面只会显示年份,而没有月份. 显示故事的页面: 现有实现首先用户填写表格,并单击"Publish",javascript会拦截请求,进行验证,这里我们只关心年和月的验证:
function validTWYear(year,month) { if(year == "") return false;//for year empty if(!isNumeric(year)) return false; var date = new Date(); var curr_date = date.getDate(); var curr_month = date.getMonth()+1; var curr_year = date.getFullYear(); if(year>curr_year) return false; if(year<1954) return false; if(year==1954 && month<5) return false; return true; } function validTWMonth(month,year) { var date = new Date(); var curr_date = date.getDate(); var curr_month = date.getMonth()+1; var curr_year = date.getFullYear(); if(year==curr_year) if(month>curr_month && month!=-1) return false; if(year==1954 && month<5) return false; return true; }
这段代码的意思是对于年字段的值而言,它不能为空,不能包含非数字的字符,不能比今年还大且不能早于1954年,如果是年份是1954,那么对应的月份应该早于5月份. 对于月字段的值而言,如果输入年份是今年,那么输入的月份的值不能大于当前的月份. 这段代码它没有任何测试案例可以寻找,另外代码的抽象程度太低,读起来很费劲,为后来的将这段逻辑移植到服务器端带来很大的不便. 关于javascript测试:Make JavaScript tests part of your build: QUnit & Rhino 如果验证都有通过,下一步就是在Controller层处理请求,这个方法是
@RequestMapping(value = "/story/new", method = RequestMethod.POST) public ModelAndView create(@Valid Story story, BindingResult bindingResult, @CurrentUser User) { ModelAndView modelAndView = new ModelAndView("new_story", "story", story); if(bindingResult.hasErrors()) { return modelAndView; } if (storyService.save(story, currentUser,request.getContextPath())){ return new ModelAndView(String.format("redirect:/story/%s", story.getId())); } return new ModelAndView("redirect:/error.jsp"); }
这段还是很清楚的,做完数据绑定之后,调用Spring内嵌支持的Validator对Story对象上所有标记Annotation的字段进行验证,并将结果保持到bindResult中,如果出现错误返回到原页面,并将错误信息一并带回. 这个过程是发生在数据绑定之后的,所以你可以猜到Story对象中一定有month和year字段. 这里必须提到Hibernate Validation Annotation , 好处在于它使得模型能够拥有自己的判断逻辑,而不需要跟其他比如UI层发生紧密的耦合,提高可重用性的同时保持自己的纯洁.但是现在Story里面的判断逻辑都是非常简单的,比如像是
@Column(name = "story_date", nullable = false) public Date getStoryDate() { return storyDate; }
然后便是storyService存储故事的代码
@Transactional public Boolean save(Story story, User currentUser,String contextPath) { story.generateStoryDate(); if (!story.valid()) { return false; } story.setUnformattedContent(HTMLTagsRemover.removeTagsIn(story.getContent())); story.generateStoryDate(); inlineTagProcessor.updateTags(story, contextPath); story.setLastModifiedBy(currentUser); story.setAuthor(currentUser); return storyRepository.save(story); }
首先,它调用story.generateStoryDate(). 这个方法是能从它的名字推出,因为story里此时记录的时间不是storyDate而是month和year,所以它需要将其转换成一个storyDate.
private Date storyDate; private String month; private String year; private Date convertToDateObject(String month, String year) { DateTime date = month.equals("-1") ? new DateTime(Integer.parseInt(year),DateTimeConstants.DECEMBER, 31 ,0,0,0,0) : new DateTime(Integer.parseInt(year),Integer.parseInt(month), 1, 0,0,0,0); return date.toDate(); } public void generateStoryDate() { setStoryDate(convertToDateObject(this.month, this.year)); }
忘了交代,如果用户在"创建新故事"的表格中没有选取month字段,那么默认会给一个值: -1. 所以为什么在convertToDateObject会是这个样子了? 因为显示这个故事的时候,需要知道是否用户当初有没有填入month,如果没有,只显示年份;如果填了,那么显示年月.这两种情况都没有跟天有什么关系, 所以此时的这段代码作者便是这样处理的: 如果用户没有在表格中填month字段,那么此时在Story对象中month的值就是-1,那就设置它的DayOfMonth是31号;如果用户有填了,那么将其DayOfMonth设置为1号. 所以你可以思考那他在显示此故事的逻辑是怎样处理的了?
Event date: ${story_view_model.dateOccurred}
在StoryViewModel中
public EscapedString getDateOccurred() { if (story.getMonth().equals("0")) return escape(story.getYear()); else return escape(new DateTime(story.getStoryDate()).monthOfYear().getAsShortText() + " " + story.getYear()); }
这段逻辑也很简单,如果getMonth()是"0",则只返回年份;如果不是则讲二则组合起来返回. 那这个"0"是从何而来了, 准备好别吐血,且看在Story中getMonth()的代码:
@Transient public String getMonth() { if(storyDate == null) return ""; if(new DateTime(storyDate).getDayOfMonth() == 31) { return "0"; } else return String.valueOf(new DateTime(storyDate).getMonthOfYear()); }
如果你将整个联合起来一想,就知道写此代码之人是如何思考的拉,如果在显示故事的时候知道是应该显示年份了还是年月一起来, 他很聪明,想到这个东西跟天没有关系,于是用天作为一个mark,同时即使在model层知道了天是否正确,那么如何让UI知道了,于是有了"0". 两个MAGIC NUMBER ,好歹你也封装下如此底层的实现细节,花了不少时间在揣测它想表达意思,后来发现是用"0"和31来串联整个流程,把我可是看吐血了. 别急还有,在Service层,调用完generateStoryDate()之后,紧接着的是:
if (!story.valid()) { return false; }
在Story中代码是:
public boolean valid() { boolean validator=checkValidYearAndTextFields(); boolean monthInFuture=checkMonthInFuture(); return (validator && !monthInFuture); } private boolean checkValidYearAndTextFields() { Validator validator = Validation.buildDefaultValidatorFactory() .getValidator(); Set> constraintViolations = validator .validate(this); return constraintViolations.isEmpty(); } public boolean checkMonthInFuture() { if (Integer.parseInt(getYear()) == currentYear()) { DateTime today = new DateTime(); if(getMonth().equals("0")) return false; return Integer.parseInt(getMonth()) > today.monthOfYear().get(); } return false; }
看起来,貌似抽象程度还可以,但是如果看checkValidYearAndTextFields()方法中,居然要一次调用Validator来验证,不知道为啥,因为CONTROLLER层已经有使用Spring内嵌提供的validator验证过了,这里再来一遍,实在是不解. 再看看checkMonthInFuture(),要检查了一遍如果输入年份是现在的年份,是否月份超出当前的月份. 于是你要可以看到神奇的数字"0",而且没有测试.

	Time Retrieving HTML	Time Elsewhere
Yahoo!	10%	90%
Google	25%	75%
MySpace	9%	91%
MSN	5%	95%
ebay	5%	95%
Amazon	38%	62%
YouTube	9%	91%
CNN	15%	85%

Tuo

Midlife Crisis as Programmer and AI

Observations from Kid Playing

My Experiement

Creativity

What connects with people is you connecting yourself

Midlife Crisis

AI

Summary

Quitting Coffee

Quitting Journey

Reflections on Quitting

Smartphone Addiction

Dopamine traps in building habits

1. Visualize too much in the starting time of a new habit.

2. Perfectionism in pursuing streaks.

Uncertainty

Feel and Being

戒断咖啡

为什么要戒断咖啡

戒断历程

戒断的思考

短视频

在某一周尝试调整一些行为来体验一下：

习惯中的陷阱

去体验和感受

NestJS+Prisma Dockerfile build optimization

Add Prisma to Dockerfile

Prisma Generate

Prisma Generate - How it works

How to run Prisma Migrate Deploy

Dockerfile optimization

1.Order your layers

2.Keep layers small

3. RUN: Combine commands together wherever possible

4. NPM/Yarn install packages

BuildKit

yarn install –perfer-offline

5. Use multi-stage builds

6. Dockerfile user and permission

Other improvement tips

Final Dockerfile

References

NestJS+Prisma Dockerfile 构建优化

Prisma上Dockerfile

Prisma Generate报错

Prisma Generate原理

如何数据迁移Prisma Migrate Deploy

Dockerfile优化

1.更好的组织构建顺序

2.减少复制到镜像每一层的文件大小

3. RUN合并命令

4. npm/yarn安装依赖

5. 使用MultiStage多阶段

6. Dockerfile用户权限

其他改善

最终Dockerfile

引用

Auto Carp Fishing Part 2 - Make it Smart and Non-Attended

Bite Alarm Approaches

Mechanism

MCU - ESP8266

Sensor - MPU-6050

Actuator - SIM800C

SIM800C Instability

5v LDO (low dropout regulator)

3.7v Li-ion Battery (Recommended)

Case Design

Demo Test

Real Test

Improvements

Prices

Source Code and Datasheets

References

Auto Carp Fishing Part 1 - Catch Big Carp In China

Nowadays Fishing Industry/Vibe

Fishing Explained

Fishing Scenario

Fishing Strategy

Swim/Location

第二个：使用单独的3.7锂电池（推荐）

更新后续一

大水面水库