M
Story
Every mathematician has only a few tricks (2020)
My dad (an engineer not a mathematician) would use Newton-Raphson[1] to solve basically any problem that wasn’t very obviously linear. When I was a kid, some of my first programming memories were my dad getting me and my brother to implement Newton-Raphson in basic on an HP85a, getting me to implement Newton-Raphson in rpn on an HP calculator, debugging my dad’s (genuinely revolting) basic program[2] which wouldn’t run but (who would have guessed?) used Newton-Raphson to compute something or other.

He basically learned the one numerical root-finder and how to evaluate basic second derivatives and he was set for life on all the problems a career in chemical and process engineering could throw at someone.

[1] https://sheffield.ac.uk/media/31988/download?attachment

[2] He learned to program in FORTRAN and lived by the maxim that a determined FORTRAN programmer can write FORTRAN in any language.

The most brilliant software developer (an EE PhD) that I have ever worked with has been using the singular value decomposition (SVD) to solve an enormous number of linear algebra and numerical computing problems in engineering software. The SVD seems to be useful for many engineering computations if you know how to apply it.
It’s the optimal low-rank approximation of a matrix in some norm-2 sense, it has those nice beautiful orthogonal factors on both, and it is often quite easy to compute. Why wouldn’t you use it, right? If you are going to have one trick, truncated SVD is a good pick.

Does he like Halko, Martinsson, and Tropp’s randomized SVD? It is pretty slick.

What are some interesting things he has used the SVD to solve?
Model reduction (for control systems), numerical precision control for matrix operations, eigenvalue computation, etc. as far as I understand.
My dad was also an Engineer. He was also a Fortran fan.

Once he asked me to explain OOP. After I explained the basics he said it was useless and never looked back.

Where OOP shines is implementing user interfaces. Most engineers doing math can ignore objects. But when a programmer has to implement a partitionable window with scroll bars and menus, constructing complicated objects built of simpler objects is a mess without OOP.
Seems like a common theme of every veteran to dress the way used to in their prime for the rest of their lives, listen to the same music, watch the same movies, etc. and to use the same belief systems as well. On the one hand, if it worked for them, why not? There's no incentive to change. Heck, it is very much the definition of conservatism. Old men who don't change is so common that it borders on proverbial.

Very rarely, however, do you see a brilliant mind like Richard Feynman, a man who was so open to new ideas and out of the box thinking. Even in old age. Seeing someone, in good faith, question what they believe in light of new knowledge is very rare. Now that is a special thing.

Fortran has OO features these days. It is nice.

I am kind of dumb and old-school so I wrote a bunch of code using macros to handle multiple precisions. If I could go back in time, I’d definitely just use object oriented code. In either case, though, “we can try a mixed precision implementation, I automatically generated single precision” is an incredibly liberating thing to be able to say as a computational scientist!

I guess as a typical engineer (not CS person type of engineer or software engineer) it is easy to think that. One might be working with machines or buildings and so on, all which require _calculation_ of processes. Those are typical cases for "just write a correct function", possibly one that takes many things into account. For such scenario OOP is truly useless and only over-complicates the matter. However, when we get to simulations, where maybe there is no known formula, or the precise calculation would be too expensive, then OOP can make sense. Doesn't have to, but could.
OOP is for problems that require complex modeling, indeed if you require just complex calculation it is useless.
  • Bayko
  • ·
  • 12 hours ago
  • ·
  • [ - ]
Chad Dad
Get your dad a copy of the classic Scientific and Engineering C++: An Introduction With Advanced Techniques and Examples by John Barton and Lee Nackman. It was written to introduce C++ to Fortran programmers using examples from scientific/mathematical domains. The fact that it is old (from 1994) makes it better suited for folks from Fortran (or other languages) since there is none of the complexity of "Modern C++" to confuse them. Check reviews on Amazon etc.
I fear not the man who practiced 1000 kicks but I fear the man who practiced one kick 1000 times
This applies to musicians. E.g. early blues guitarists only had a few licks but they played them so many times they honed them into perfection.
Relatedly, I've found Newton-Raphson is a great example of an algorithm where Knuth's "I have only proven it correct, not tried it" rears its head very prominently. The obvious implementations can work flawlessly on toy examples and then fail miserably on real-world examples.
There's a reason why numeric analysis is still actively studied by research mathematicians. If we could just throw something as simple as newton's method at any nonlinear problem, we'd only need people to learn this once in school and everyone could solve everything.
But I'm not sure I'd recommend going into the field. There's something demoralizing about doing research on something which already has dozens of valid and successful methods, of which you are trying to create a slightly more optimized version.
I am so into optimizing fast polynomial multiplication I assure you there is nothing that will demoralize me from creating a slightly more optimized version.
It's not for a field for everyone.

But that "slightly more optimized version" may mean "one that does not quietly produce disastrously incorrect results for some input values".

I should probably qualify my statement more. There are certainly new and interesting problems in numerics. And even going from say O(n) convergence to O(n log n) convergence can lead to whole new classes of problems you can solve. I don't want to discourage anyone who loves numerics.

What I was trying to say was that as a graduate student you might be given a problem that already has many really good and smart solutions and be essentially told to find a better solution than all of these. How this goes will depend a lot on the specific problem, your advisor, etc.

Yeah for that you need Euler’s method… I mean of course Runge-Kutta. … By that I’m of course referring to rk4. … I mean, you have a point.

Joking aside I think having a few basic numerical methods like say Newton, rk4, brent root, monte carlo simulation etc in your general toolbox of techniques you know how to do can make you unreasonably effective in a wide range of situations. Just yesterday I had to solve a problem in a relatively small space so I first used a brute force method to check all the feasible solutions and having got the full list of actual solutions out, figured out the analytical solution. It meant I could be very confident that my analytical solution was correct when I had it.

If you know some basic composition methods, Euler, RK and all the higher order methods can be easily constructed as you need them. But there are still many applications where you for example want to use symplectic methods instead. If you know about those+composition, you can at least solve more or less all of classical newtonian dynamics. But if you go to quantum mechanics or field theory (or quantum field theory), you need to enter a whole other world of numerics to get real world usage out of it.
I would also toss differential evolution in the ring as a widely applicable and easy to implement technique.

https://en.wikipedia.org/wiki/Differential_evolution

Nobody has enough memory or patience for third order derivatives so Newton's method (aka Newton Raphson) it is.
The modern world of autodiff actually makes real second and third order derivatives fairly cheap to compute.

The stuff that is actually used most commonly, only uses first order derivatives though (gradient descent, Levenberg-Marquardt, Kalman filters...)

I could be wrong, but "memory" and "patience" sounded like they were referring to machine memory and patience waiting for a slow algorithm, which is what you would expect from any derivative more involved than a Jacobian for nontrivial problems, even when doing tricks like vjp or vhp.
I think this is true for engineers as well! I enjoy getting to know the "theme" of my favorite coworkers over the years. There was:

* The fellow who always looked for the simplest hack possible. Give him the most annoying problem, he'd pause, go Wait a minute! and redefine it to have a very easy solution. He typed very slowly, but it didn't really matter.

* The one who truly loved code itself. He would climb mountains to find the most elegant, idiomatic way to express any program. Used the very best practices for testing, libraries, all that. He typed very fast.

* The former physicist who spent all his time reading obscure mailing lists on his favorite topics. His level of understanding of problems in his domains of interest was incredible.

I could go on and on! It's such a fun taxonomy to collect. All of these friends were marvelous at solving their particular flavor of problem.

As for myself, I like to think that my "trick" is to spend a long time poking at the structure of a problem. Eventually the solution I was looking for doesn't matter anymore, but the tools I developed along the way are pretty useful to everyone!

Here are a few more.

* The (brilliant) infrastructure engineer who described his modus operandi as 'I read stuff on Reddit and then try it out.' This engineer is now worth, as a conservative estimate, in the neighborhood of $50 million. So maybe more of us should be doing that.

* Another infrastructure engineer, also very effective, who made a habit of booking an external training session (sometimes a series, weekly) for how to set up and integrate every piece of technology we used.

* An engineer (this one is quite famous, and we have only interacted professionally a few times) who simply wrote the best comments in the world. Every method was adorned with a miniature essay exploring the problem to be solved, the tradeoffs, the performance, the bits left undone. I think about those comments quite often.

As an addendum, though, I will say that the best engineers overall all shared a trait - they kept trying things until they got something working. That alone will take you pretty far.

“The most successful people have failed more times than you have tried”
Success is relative. If the goal is to never fail, never try is the best strategy.

Also the most sure path to finish in the 1% wealthiest is to start in its network.

When the game is set to make 99% of players considered as losers in its own terms, the best strategy to have fun at scale is to not care about the highlighted goal. Keep awareness of how rules actually apply, take shortcuts if it feels safe and preferable, always respect human dignity even when nasty players try to make a dirty move agaisnt you, don't let the lowering bare of hate infect one more player.

I am having trouble thinking of situations where the only good outcome is being the top 1%. There are, to be sure, often power-law benefits to being exceptionally good, but to use your own example, being in the top 5% or 10% of wealth is nothing to sneeze at. It's all a spectrum.

With that said, I think you make some good points about the perspective people should have about competitive goals. Most contests are not strict meritocracies, and you are in for a lot of frustration if you assume that only effort matters when other slide by on their structural advantages. I would prefer to reframe the goal as 'work hard on what you have control over, as long as it provides some amount of benefit, and as long as you still care about the outcome.'

> always respect human dignity even when nasty players try to make a dirty move against you

What a gem of a quote. A great way to avoid becoming a bitter person.

>Success is relative. If the goal is to never fail, never try is the best strategy.

This is dangerously not true, if you never try, then you are guaranteed to fail to live up to your potential, which is one of the greatest failures of all.

Yes. I like to think that all of the people above could have solved most of the same problems, albeit in wonderfully different styles, but what really guaranteed success was a commitment to just keep at it.

Edit to add: Still, the styles matter a lot! For one thing, they greatly influence which problems each person is interested in. Also, the style you solve problems with colors what your final output looks like, which is perhaps more obvious in engineering than in mathematics.

For me, it's tracing code/pipelines to figure out how a result was produced, typically in the context of that result being wrong somehow. Go To Definition is the most useful function in any editor.

I'm always surprised by how frequently colleagues don't think to do this and are left helpless.

This reminds me of my further theory that everyone needs one 'heavy' and one 'light' technique. The 'light' technique is something that often works well as a heuristic and can be an effective unit of iteration. The 'heavy' technique is something that you can fall back on in difficult cases, something that can reliably solve hard problems, even if it's slow.

Sometimes the heavy technique is: just ask someone else. ;)

> Sometimes the heavy technique is: just ask someone else. ;)

For a lot of people I know, this is the light technique!

You jest, but that's how my sister gets through life, and it's always fascinated me.

She's incredibly intelligent, but more importantly she's a phenomenal social networker. She always has someone to call to ask about any question or solve any problem that comes up in life, and she's great at connecting these people with each other when they have problems of their own - so they all want to help her with whatever she needs, just to gain access to a pool of people they themselves can talk to.

What do you do with a skillset like that? I honestly don't know - something in leadership, probably, something where finding the right people and setting them to work is the most important skill of the job.

That wasn't in jest. I worked in a place where this was a norm. Nothing was properly documented, instead everyone would just ask and answer questions on chats; somehow, this actually kept velocity high.

Found it really hard to adjust to that. I'm the kind of person that prefers to research things on my own, find hard references and understand context. But there, this was the wrong approach.

for those, llms are a godsend
I have both felt and seen this at work and I would add to this the meta-technique of binary search. Once it is added to your light and heavy technique you can solve what seems intractable at first glance faster than many people can even orient to the problem.
A related meta-technique is "starting at both ends and meeting in the middle". It's kind of like the inverse of binary search in some sense, isn't it?
  • ant6n
  • ·
  • 15 hours ago
  • ·
  • [ - ]
For me the heavy technique is integer linear programming.

I’m not a software developer anymore.

Likewise. I don't always do this, but for problems that cost me much time or effort, I like to try to make sure that, if I wanted to reproduce a bug or problem, I'd know exactly how to write it.

Writing and understanding working correct software is, it turns out, a rather different skill from that of writing and understanding broken (or confusing) software. I'd also wager good money that the latter skill directly builds the former.

Another example: debuggers.

It's amazing that a lot of new developers don't know how to use them at all! I'm not even talking about using the command line gdb, but just the basic "Set Breakpoint" feature and attaching to processes.

[dead]
My impression from math lectures as a CS student was, that you have to know the tricks, otherwise you just sit there dumbfounded, not knowing how you can make progress. Obviously, there will also be some kind of mathematical understanding of things at play, but a lot of the math stuff is pattern recognition and applying tricks. If you don't know them, it becomes very frustrating. I've not ever seen a way of teaching math though, that focuses on teaching learners these tricks or techniques. They always fell out of things on the side or the teaching person assumed you know it, or was surprised, that the learning people were not able to see it, or able to apply some trick. Or they just didn't care, as long as a small portion of the learners somehow were able to solve it, because then it obviously cannot possibly have been their way of teaching, but other students must have been lazy...
Feynman wrote in his autobiography that much of his success came from having different mathematical tricks than most of his peers. So when they were stuck, he could sometimes make progress.
Interestingly, one of his famous tricks of computing complex integrals by parameterizing[1] and then differentiating under the integral is known as “Feynman’s trick” in his honour spite of it having been invented by Euler over 250 years before.

[1] https://zackyzz.github.io/feynman.html

  • tux3
  • ·
  • 14 hours ago
  • ·
  • [ - ]
A popular naming convention in Mathematics is to name things after the second person to discover them. The first person is Leonhard Euler.
He had many tricks in limited area. All his his tricks were limited to classical calculus and nineteenth-century mathematics. He didn't do anything fancy.

His style was always the same, he just mastered it really well.

That is the actual trick. To master one trick so well that you can spot anywhere its hiding. Unfortunately, this takes a long time, which is why most mathematicians can only do it for a couple of tricks in their lifetime
Feynman used to read his own books. When asked he said, "it's all in here". He used to revise and refresh his own understanding.
I remember annoying one of our professors at the university. Whenever he was discussing a problem and found me falling asleep, he called my name. I woke up, said Chinese remainder theorem and with like 90% success rate it solved the problem handily. Yes, it was an Algebra class. We were still surprised how well it worked.
My lecturer was once unable to solve a problem at the blackboard. After trying for some time, he said he needed a short break to go back to his office to look at his notes. He brought the notes, and there he found a hint he had written for himself: "Use a trick"
  • brap
  • ·
  • 13 hours ago
  • ·
  • [ - ]
For programmers I would say graphs or at least thinking in graphs is a common one.

Some would say SAT is also a handy trick but I’ve personally never used it.

SAT, SMT, ILP, MILP.
One of the answers links https://www.tricki.org/, which describes itself as a 'Wiki-style site with a large store of useful mathematical problem-solving techniques.' no longer maintained, but looks neat.
There is a joke in applied mathematics that we’re like Taco Bell. We all use the same six ingredients, mixing them in different ways.

For myself, I’ve found several techniques I use over and over again. Some of this is a “when you’re a hammer, everything looks like a nail.” But fundamentally there are only a handful of ideas. One professor of mine once said the only groundbreaking result in the past few decades was compressive sensing.

  • fizx
  • ·
  • 9 hours ago
  • ·
  • [ - ]
- The hard part of a compiler is the parser. Find an existing parser, then emit using that language's web templates.

- Your DB query is better in an inverted index.

- Consider your data locality carefully.

In my world every triangle is a right triangle and I my hammer is the Pythagorean Theorem.
From a comment:

> These methods can be combined. First generalize the problem, making it more complicated. Then simplify along a different axis. – Stig Hemmer

Very relevant to software design too.

For most of my computer science PhD the “trick” was just to get the inductive definition to work, and then how to tweak it for the next paper. Or, get enough structuret we can do an “abstract nonsense” proof[0].

[0]:https://ncatlab.org/nlab/show/category+theory#AbstractNonsen...

One surprise of reading this thread was learning that Terry Tao is an active user of Math Overflow!
I like the trick of adding zero to an equation.
A mathematician tricks.always. There is no point to untrick the trickster. Misuse of your time
I do various things where for example I need to convert a linear movement of a control knob to a non-linear value.

Quite often I'll look at it and go "aha, it's exponential subtracted from a constant", and then go "aha, that doesn't quite work, it's a polynomial", and then fiddle with that for twice as long.

Eventually, "meh, sod it, I'll just sketch out a graph of what I want on paper, turn it into a lookup table, and LERP for the correct value".

It's amazing how quickly you get a sense of "I just need to bend that in a little so that if these are both up full it doesn't go absolutely mental", and plan your tables accordingly. Also ROM is cheap these days.

[dead]
  • ·
  • 20 hours ago
  • ·
  • [ - ]
  • NedF
  • ·
  • 20 hours ago
  • ·
  • [ - ]
[dead]