Log-Structured Merge Tree with level-based compaction

2017-01-10  |   |  computer science  

It is surprisingly hard to find a good explanation to level-based compaction of a Log-Structured Merge Tree. It turns out that it is best explained in LevelDB's documentation. You can find the (html) details here.

This blog post is a collection of key concepts I did not grasp initially. Sort of a mental note for me. It should bring you nicely from standard size-based compaction LSM to level-based compaction.

Levelled LSM structures are useful as they limit greatly the number of files to access when reading a given key. nbrOfLevel0Files + (n-1) where n is the number of levels.

You still have levels but level 1 and above have different behaviors:

  • there is the in memory level + append only log (non ordered)
  • there is level 0 which behaves like a normal LSM level (each file is ordered but has overlapping keys)
  • level 1 and above have files containing non overlapping keys

I call segment, a given file at a LSM level (containing ordered keys), it is called sstable usually. Why do I call it segment? I come from Lucene and I have read Dune and know of sandworms. Plus segment is a much nicer word than sstable :)

The non-overlapping ranges for a given level L are not fixed in stone and are recomputed each time a compaction from level L to level L+1 occurs. Big ahah moment for me.

When a level L is merged into level L+1 (for L >= 1), one segment of L and all overlapping segments of L+1 are read. New segments are created at level L+1 from this data and a new level L+1 is created from these new segments and the existing non-overlapping ones. When compaction is done, the manifest (reference of segments) is updated and old segments are deleted. From that data, new segments at a given level are created based on:

  • size (e.g. every 2 MB)
  • as soon as the key range of a given segment overlaps with more than 10 segments at L+2

Tombstones are kept around until the last level (to make sure we hide the possibly older values in higher levels). They claim that they remove tombstones for a given key if no higher level has a segment with whose range overlaps the current key but that looks like a minor optimisation.

In LevelDB, the max size of a level is 10^L MB (e.g. 10 MB for 1, 100MB for 2 etc). Levels do increase in size exponentially though each segment is of fixed size (at least not exploding).

All this compaction only involves sequential reads and sequential writes (when done right).

I'm well aware that many improvements have been built atop this initial approach but they all rely on you understanding this first cornerstone improvement :)

Cool stuff.

Devoxx CfP - the art of tricking yourself

2017-01-05  |   |  conference  

I noticed that the Devoxx France call for paper (CfP) application was influencing my votes. Sneaky one!

Jeff Atwood's tweet made me rethink about something I noticed

When we designed Stack Overflow I intentionally put author at bottom: you should read the actual content before "deciding" credibility

In the CfP, you review a proposal, you first see:

  • the title and type of proposal (conference, Tools in Action etc)
  • then the abstract
  • then the private message to the committee

At this stage, you will have to scroll down to see more, especially if you are reviewing on a tablet like I do. Your brain will make a pre-judgement based on the title and abstract like any attendee. It will later absorb the private message but things are already too late (kind of).

It's only after scrolling that you will see who the person is and what qualification he or she has.

I don't know if Nicolas did it on purpose but it has brilliant side effects:

  • You will value good titles and good abstracts over good back channel info
  • You will alter your judgment based on the person qualification and fame last. Congnitive research seems to indicate that your reptilian judgement favors the first data much more.
  • This is a nice trick to favor subjects over rock stars

What's even more brilliant is that even if I'm concious of this, the trick still works :)

A good presentation is a mix of good subject, good content and good presenter. I do think good presenters are key but besides fame or first hand experience, it is the hardest to judge. What I love about the way the CfP app does it, is that it is a harder process for me to either:

  • overcome a just ok proposal from a famous speaker
  • overcome a good proposal from a unknown quantity speaker

I still do alter my note of course based on who proposes: that's part of the magic equation. But certainly less than if the name was first.

Now I get how Devoxx France "encourages" new speakers.

Why do we have corporate income tax anyways?

2016-08-30  |   |  economy  

There is a lot being written about corporate tax optimisation/evasion these days both in Europe and in the US. This begs for a more general question, why do we tax corporate profits? I had the debate with a friend this summer and led me to research the topic.

There is a very interesting paper on this subject as well as an analysis of the distortion of corporate tax. Here is my summary.

Why (not) a corporate income tax

The main arguments - to me - in favor or against corporate income tax are summarized in the following paragraphs.

A corporation benefits from common infrastructures (highways, social security etc) and thus must pay its due to Society.

However a corporation is (in the end) always owned by individuals which themselves do pay taxes to finance common infrastructures. This is argued by some as leading to a double taxation.

A corporation can be owned by foreign investors: better tax these guys via corporate taxes rather than the folks that actually vote for us.

A corporate tax leads to some sort of pre-tax of the foreign investor by virtue of lesser dividends.

Individuals would feel it to be unfair if they were to pay for all taxes while corporations are making plenty. Of course, individuals do indeed pay it all in the end whether they see it or not but it looks like it's a hard notion to grasp for most.

A corporation bringing the benefits from a foreign subsidiary can deduct from its dividend tax the actual income tax it paid in the subsidiary's country. This essentially erase the foreign income tax assuming the rate is lower than the domestic dividend tax rate. These treaties are here to avoid double taxation and lessen the burden of an income tax.

Without corporate income tax, personal income tax diminishes as individuals find ways to "incorporate" their revenues to avoid taxation.

And of course the cynical view is that governments are addicted to spending and they need more fresh cash than a junkie needs dope (this argument is not in the paper for obvious reasons).

What distortions does it cause

Again this is a personal cherry picking from the paper. What is interesting is that this paper is based mostly on studies of the EU, not the US as it is often the case.

Small companies are often offered lower tax rates, to compensate for market failures. It would be better to use a separate explicit mechanism (e.g. direct aids) to compensate for them. As it is, different tax rate brackets create a disincentive to grow.

A European study shows that the more corporate income tax, the lesser wages are: for every additional euro of corporate income tax, wages are reduced by 0,92 in the long run. Income tax is not good for your salary apparently :)

As explained in the previous section, a corporate income tax lower than the personal income tax leads to a shift from personal to corporate taxes. People (e.g. entrepreneurs) optimise and "incorporate" their income. This is one of the few arguments that encourages a higher income tax.

Income tax influences where an international company opens foreign subsidiaries (a 1% point income tax increase, decreases the change of the subsidiary being opened by 3,96%). Ouch!

Same for foreign investment: a 1% point tax increase, decreases foreign investment by 2.9%.

And finally profit shifting. Profit shifting is what big international companies are accused of these days (Apple, Google, Starbucks, Ikea etc). One study estimates that due to this phenomenon, a 1% point increase in tax rate leads to a loss of 17,2% of the planned extra tax collection. I'm personally skeptical of the averages. We cannot consider this phenomenon by mean nor median: I imagine a company engaging in such activity would do it in an all or nothing fashion.

What's the take away?

Tax is hard, you touch one button and unexpected things move all over the place. Be careful of tax that go to 11, you might become deaf... and sterile ;) More seriously, this paper has been hard to find but knowing about all this will make you a better citizen.

Read the paper, there is a lot more to it

I had to summarize, cherry pick and cut corners to keep this entry short. Go read the paper which is easy to read (except in some specific areas), goes in greater details and cite all of its sources. And above all it is very interesting !

I put a copy of this paper here since it disappeared from its original location. This paper is copyright the European Commission and written by the staff of the European Commission's Directorate-General for Taxation and Customs Union (Gaëtan Nicodème in particular).

Homebrew formulae for Mutt with sidebar and trash patches

2016-03-15  |   |  Mac OS X   tool  

Homebrew's Mutt formulae is in a bit of a disarray. I can't blame them as Mutt has a bunch of not quite maintained patches not quite fully compatible with one another.

The problem is that some of these patches are very very useful. I have created a tap to maintain Mutt with the two key patches I use:

  • sidebar
  • trash

At the time of writing, it uses Mutt 1.5.24 but I might update it. To use the formulae, do:

brew tap emmanuelbernard/mutt
brew install emmanuelbernard/mutt/mutt

// or alternatively

brew install https://raw.githubusercontent.com/emmanuelbernard/homebrew-mutt/master/Formula/mutt.rb

I personally build them with the following options

brew install emmanuelbernard/mutt/mutt --with-sidebar-patch --with-trash-patch --with-gpgme --with-s-lang

s-lang supposedly has better support for color schemes like Solarized.

You can find the code at https://github.com/emmanuelbernard/homebrew-mutt.

Synology rsync and non standard ports

2016-03-09  |   |  infrastructure  

Recently running rsync to my Synology diskstation stopped working. I had just changed my default SSH port to a non standard one. Learn how to fix it.

Synology recommends to change the default SSH port to a non standard one. What they forgot to tell you is that it will break your ability to rsync into the machine. Here is a way to fix it.

The workaround way

One way to fix it is to use the --rsync-path=/usr/syno/bin/rsync option.

rsync -avz --rsync-path=/usr/syno/bin/rsync from-dir/ synology:/volume1/Backups/to-dir

Since you use a non standard SSH port, make sure to also update your .ssh/config file to point to the right one.

Host synology
    User alice
    Hostname # my synology IP
    Port 911 # my new SSH port and a nice car

That works around the Synology quirk but it requires to update all your rsync scripts.

The proper way

Log to the web management console and open the Backup & Replication application. Select Backup Services and update the SSH encryption port to match your new SSH port, in my example 911. Note that for some reason the UI forbids certain port numbers. Make sure to use a non restricted number for your SSH port in the first place or use trial and error.

The SSH port can be changed in the Control panel, Terminal & SNMP.

Asciidoc folding for Vim

2016-02-11  |   |  tool  

I just learned about the ability to fold in Vim. For mere mortals, it means hiding parts of the file.

Here is a code to put in your .vimrc to allow folding for Asciidoc(tor) files. It folds asciidoc files at section boundaries and use nested folds for subsections.

"" Fold Asciidoc files at sections and using nested folds for subsections
" compute the folding level
function! AsciidocLevel()
    if getline(v:lnum) =~ '^== .*$'
        return ">1"
    if getline(v:lnum) =~ '^=== .*$'
        return ">2"
    if getline(v:lnum) =~ '^==== .*$'
        return ">3"
    if getline(v:lnum) =~ '^===== .*$'
        return ">4"
    if getline(v:lnum) =~ '^====== .*$'
        return ">5"
    if getline(v:lnum) =~ '^======= .*$'
        return ">6"
    return "="
" run the folding level method when asciidoc is here
autocmd Syntax asciidoc setlocal foldexpr=AsciidocLevel()
" enable folding method: expression on asciidoc
autocmd Syntax asciidoc setlocal foldmethod=expr
" start with text unfolded all the way
autocmd BufRead *.adoc normal zR
autocmd BufRead *.asciidoc normal zR
" TODO following does not work as folding is lost up reloading
" autocmd Syntax asciidoc normal zR

I'm sure it can be improved - I'd love to fold blocks as well - but that's a start.

Here are a few commands to remember to fold in Vim

  • zo: open a fold at cursor
  • zO: open all folds down at cursor
  • zc: close a fold at cursor
  • zC: close all levels of folds at cursor
  • za: / zA: toggle fold
  • zm: close fold by one level across the file
  • zM: close all folds across the file
  • zr: open fold by one level across the file
  • zR: open all folds across the file
  • zj / zk: move to next / previous fold
  • [z / ]z: go to begining / end of the fold

Happy folding.

Adding color to Maven output

2015-07-27  |   |  tool  

Maven is quite verbose. Finding the useful information when the test fails requires you to squint eyes. Unless, you bring some coloring to the massive Maven output.

The state of color output in Maven is still quite messy. Just look at Arnaud's blog to see how non user friendly that is.

Enter maven-color

Enter Jean-Christophe and its Maven Color project. The goal is to bring colorized maven console in an easy and cross platform way.

It's relatively easy to install (check the README), and is even easier on Mac OS X

brew tap jcgay/jcgay
brew install maven-deluxe

From there you might need to unlink your brew maven install.

Usually, you are done. Well, not if like me you use CheckStyle.

SLF4J beam crossing

Unfortunately for me, it was failing on Hibernate OGM. The problem is that the CheckStyle plugin is compiled with the Maven 2.x version of SLF4J aka an old one.

This leads to funky errors like - in color mind you:

Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:checkstyle (check-style) on project hibernate-ogm-core: Execution check-style of goal org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:checkstyle failed: An API incompatibility was encountered while executing org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:checkstyle: java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;Ljava/lang/Throwable;)V
realm =    plugin>org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1
strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
urls[0] = file:/Users/emmanuel/.m2/repository/org/apache/maven/plugins/maven-checkstyle-plugin/2.12.1/maven-checkstyle-plugin-2.12.1.jar
urls[1] = file:/Users/emmanuel/.m2/repository/org/slf4j/slf4j-jdk14/1.5.6/slf4j-jdk14-1.5.6.jar
urls[2] = file:/Users/emmanuel/.m2/repository/org/slf4j/jcl-over-slf4j/1.5.6/jcl-over-slf4j-1.5.6.jar
urls[3] = file:/Users/emmanuel/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar
urls[4] = file:/Users/emmanuel/.m2/repository/org/codehaus/plexus/plexus-interactivity-api/1.0-alpha-4/plexus-interactivity-api-1.0-alpha-4.jar
urls[5] = file:/Users/emmanuel/.m2/repository/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.jar
urls[6] = file:/Users/emmanuel/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar
urls[7] = file:/Users/emmanuel/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar
urls[8] = file:/Users/emmanuel/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.0/maven-reporting-api-3.0.jar
urls[9] = file:/Users/emmanuel/.m2/repository/org/apache/maven/reporting/maven-reporting-impl/2.2/maven-reporting-impl-2.2.jar
urls[10] = file:/Users/emmanuel/.m2/repository/org/apache/maven/doxia/doxia-core/1.2/doxia-core-1.2.jar
urls[11] = file:/Users/emmanuel/.m2/repository/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar
urls[12] = file:/Users/emmanuel/.m2/repository/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar
urls[13] = file:/Users/emmanuel/.m2/repository/org/apache/httpcomponents/httpclient/4.0.2/httpclient-4.0.2.jar
urls[14] = file:/Users/emmanuel/.m2/repository/org/apache/httpcomponents/httpcore/4.0.1/httpcore-4.0.1.jar
urls[15] = file:/Users/emmanuel/.m2/repository/commons-codec/commons-codec/1.3/commons-codec-1.3.jar
urls[16] = file:/Users/emmanuel/.m2/repository/commons-validator/commons-validator/1.3.1/commons-validator-1.3.1.jar
urls[17] = file:/Users/emmanuel/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar
urls[18] = file:/Users/emmanuel/.m2/repository/commons-digester/commons-digester/1.6/commons-digester-1.6.jar
urls[19] = file:/Users/emmanuel/.m2/repository/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar
urls[20] = file:/Users/emmanuel/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.4/doxia-sink-api-1.4.jar
urls[21] = file:/Users/emmanuel/.m2/repository/org/apache/maven/doxia/doxia-logging-api/1.4/doxia-logging-api-1.4.jar
urls[22] = file:/Users/emmanuel/.m2/repository/org/apache/maven/doxia/doxia-decoration-model/1.4/doxia-decoration-model-1.4.jar
urls[23] = file:/Users/emmanuel/.m2/repository/org/apache/maven/doxia/doxia-site-renderer/1.4/doxia-site-renderer-1.4.jar
urls[24] = file:/Users/emmanuel/.m2/repository/org/apache/maven/doxia/doxia-module-xhtml/1.4/doxia-module-xhtml-1.4.jar
urls[25] = file:/Users/emmanuel/.m2/repository/org/apache/maven/doxia/doxia-module-fml/1.4/doxia-module-fml-1.4.jar
urls[26] = file:/Users/emmanuel/.m2/repository/org/codehaus/plexus/plexus-i18n/1.0-beta-7/plexus-i18n-1.0-beta-7.jar
urls[27] = file:/Users/emmanuel/.m2/repository/org/apache/velocity/velocity-tools/2.0/velocity-tools-2.0.jar
urls[28] = file:/Users/emmanuel/.m2/repository/commons-chain/commons-chain/1.1/commons-chain-1.1.jar
urls[29] = file:/Users/emmanuel/.m2/repository/dom4j/dom4j/1.1/dom4j-1.1.jar
urls[30] = file:/Users/emmanuel/.m2/repository/sslext/sslext/1.2-0/sslext-1.2-0.jar
urls[31] = file:/Users/emmanuel/.m2/repository/org/apache/struts/struts-core/1.3.8/struts-core-1.3.8.jar
urls[32] = file:/Users/emmanuel/.m2/repository/org/apache/struts/struts-taglib/1.3.8/struts-taglib-1.3.8.jar
urls[33] = file:/Users/emmanuel/.m2/repository/org/apache/struts/struts-tiles/1.3.8/struts-tiles-1.3.8.jar
urls[34] = file:/Users/emmanuel/.m2/repository/org/apache/maven/shared/maven-doxia-tools/1.4/maven-doxia-tools-1.4.jar
urls[35] = file:/Users/emmanuel/.m2/repository/commons-io/commons-io/1.4/commons-io-1.4.jar
urls[36] = file:/Users/emmanuel/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar
urls[37] = file:/Users/emmanuel/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar
urls[38] = file:/Users/emmanuel/.m2/repository/org/codehaus/plexus/plexus-resources/1.0-alpha-7/plexus-resources-1.0-alpha-7.jar
urls[39] = file:/Users/emmanuel/.m2/repository/org/codehaus/plexus/plexus-utils/3.0.15/plexus-utils-3.0.15.jar
urls[40] = file:/Users/emmanuel/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.19/plexus-interpolation-1.19.jar
urls[41] = file:/Users/emmanuel/.m2/repository/org/codehaus/plexus/plexus-velocity/1.1.8/plexus-velocity-1.1.8.jar
urls[42] = file:/Users/emmanuel/.m2/repository/com/puppycrawl/tools/checkstyle/5.7/checkstyle-5.7.jar
urls[43] = file:/Users/emmanuel/.m2/repository/antlr/antlr/2.7.7/antlr-2.7.7.jar
urls[44] = file:/Users/emmanuel/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.3/commons-beanutils-core-1.8.3.jar
urls[45] = file:/Users/emmanuel/.m2/repository/com/google/guava/guava-jdk5/14.0.1/guava-jdk5-14.0.1.jar
urls[46] = file:/Users/emmanuel/.m2/repository/org/apache/velocity/velocity/1.5/velocity-1.5.jar
urls[47] = file:/Users/emmanuel/.m2/repository/commons-lang/commons-lang/2.1/commons-lang-2.1.jar
urls[48] = file:/Users/emmanuel/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar
urls[49] = file:/Users/emmanuel/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
Number of foreign imports: 1
import: Entry[import  from realm ClassRealm[maven.api, parent: null]]

-> [Help 1]

There is a relatively easy fix. You can force the SLF4J version of CheckStyle in your plugin dependencies.


Now I can get spanked by CheckStyle in color!

If like me you drawn in Maven outputs, go give a try to Maven Color. And many thanks to Jean-Christophe for his help in solving the CheckStyle of death problem.

Moving to HTTPS

2015-05-09  |   |  infrastructure  

I have decided to move this website to HTTPS. Let me know if you see anything fishy, cannot access it or if the RSS feed no longer updates. Why? With my country's government going completely stupid, I decided to play the nut game, I can be pretty stupid too.

In its infinite wisdom, the French government has decided to enable mass surveillance of the French people without oversight of the justice branch. Yes I know WTF?!

Anyways, in order to make things simpler for them, I have decided to move all of my web properties behind HTTPS (HTTP over TLS). Do I have anything to hide? Nope, I just want to increase universe entropy and cracking our communication a bit more challenging.

My certificate fingerprints are:

  • SHA1: 15 3E 2E 67 39 28 20 28 6E F6 CF F5 E1 92 AB 07 72 32 72 90
  • MD5: 0B 2D 49 61 DE EA E1 D7 11 92 BA 3D 4F BD 73 E2

If you see something else then, you are being intercepted. Read more on secure connection fingerprints.

SPF for dummies - how to fight spams

2015-03-26  |   |  infrastructure  

In this post, you will learn everything you need about SPF (Sender Policy Framework), what it means for your emails and how / if to set it up for your domains.

What is it for and should I care?

This is a standard that helps reduce spam. Each domain lists in one DNS record the list of servers that are allowed to send emails for that domain. So when an email provider like Gmail sees an email sent from an address @example.com but coming from a server not listed in the SPF record, it knows it is likely a spam.

Conversely, it is important to set such a record to avoid your emails to be considered spam. More and more email providers consider domains without SPF record as more suspicious than others. Even if you domain does not send emails, you should set a SPF record, this will prevent spammers from faking emails from you.

How does it work ?

As owner of a given domain, you will tell the world which servers you will send emails from. That is typically your SMTP server.

In practice, this is a TXT entry in the DNS records of your domain. Something like

"v=spf1 a include:_spf.google.com ip4: -all"

v=spf1 is the protocol, all entries start with this.

a or generally a:example.com says that all IPs listed in the DNS A record of the domain should be able to send emails for that domain. If the domain is not specified (a), then it is the domain for which the TXT DNS record lives.

ip4: means that IP is allowed to send emails for your domain. You can assign ranges as well. Likewise, there is an ip6 syntax.

include:_spf.google.com means that you should consider the SPF rules stored in the DNS entries of _spf.google.com. This is very useful if you use Google Apps / Gmail or send emails from another domain's SMTP server. Rules can be composed, so you can have explicit ips, a and includes in the same SPF entry.

There are also mx and ptr entries but I won't go into details.

Finally, you need to decide what to do when the rules don't match. That's where the all mechanism comes into play:

  • -all means that servers that do not pass the rules should be considered spammers
  • ~all means that servers that do not pass the rules should be considered spammers but we are not 100% sure so let them pass but be suspicious
  • ?all means that servers that do not pass the rules should be considered neutral (i.e. they may be legit or not)

If your list is exhaustive, use -all to lock things down. If people sending emails from your domain might use their own SMTP server, use ?all. ~all is for chickens ;)

You can get a formal description of all the options at openspf.org.

You can lookup the SPF entries for any domain by typing

nslookup -q=TXT example.com

This is useful before you add an include clause.

A few concrete examples

My domain never sends any email: "v=spf1 -all".

I use Gmail / Google Apps to send emails from that domain: "v=spf1 include:_spf.google.com -all".

I use Gmail and Red Hat's SMTP: "v=spf1 include:_spf.google.com include:redhat.com -all".

I use the server hosting example.com and Red Hat's SMTP: "v=spf1 a include:redhat.com -all" (this is a short hand for "v=spf1 a:example.com include:redhat.com -all"

People sending email as other-example.com use various SMTP servers but for sure and it's corresponding IPv6 2001:db8::df4:cd23 and example.com's SMTP servers: "v=spf1 ip4: ip6:2001:db8::df4:cd23 include:example.com ?all"

A note on TXT DNS entry, to use (I think) the bind syntax, here is how it looks like

# let's say it's the DNS zone of example.com
# some domain
@ 10800 IN A
# some subdomains
awesome 10800 IN CNAME example.com.
blog 10800 IN CNAME example.com.
# looks like these guys use Google Mail
@ 10800 IN MX 10 aspmx.l.google.com.
awesome 10800 IN MX 10 aspmx.l.google.com.

# Here is our subject
# you need one entry per domain and subdomain sending emails
@ 10800 IN TXT "v=spf1 a include:_spf.google.com -all"
awesome 10800 IN TXT "v=spf1 a include:_spf.google.com -all"
# blog never sends emails
blog 10800 IN TXT "v=spf1 -all"

Conclusion and a few recommendations

I prefer domain names over IPs so I use a or mx entries. As much as I can, I use include and delegate the list to the real guys.

Google and others use special subdomains like _spf.google.com to host their SPF rules. This is useful to separate different ruleset but bind them together in your primary domain via an include rule. If you are one of them, you probably don't need my blog entry in the first place :)

Remember that servers now have IPv6 addresses and that Google and other have already IPv6 infrastructures in place. Don't forget them, I had some emails denied because I was missing it and my server communicated with Google from an IPv6.

My take out is simple: if you own a domain and send emails, add a SPF entry. It's relatively simple and the examples I gave you should get you a long way already.

PS: I am relatively new to this domain, feel free to correct me in the comments, if I made a mistake.

Get notified when terminal commands end

2015-03-26  |   |  tool  

Here is a tiny little tool that will speed up the multi-tasking life of terminal users: be notified when a command finishes.

How many times have you started a command in the terminal to realise that it will take a while? How many times did you then move to emails or twitter "in the mean time"? How many times have you forgotten about it and read our twitter feed for 30 minutes aka 25 minutes longer than the actual command?

This small tool solves that problem.

# Notify you when a task is done
# $ notify mvn clean install
# runs 'mvn clean install'
# notify you when it's done
# A notification is sent upon build completion if your OS supports it:
# - on Mac OS, install Growl and grownnotifier
# - on Linux, install send-notify
# Released under the WTFPL license version 2 http://sam.zoy.org/wtfpl/
# Copyright (c) 2010 David Gageot
# Copyright (c) 2011 Sanne Grinovero
# Copyright (c) 2010-2015 Emmanuel Bernard

say() {
    if [ `uname -s` == "Darwin" ]; then 
        # On Mac OS, notify via Growl
        which -s growlnotify && growlnotify --name "Command line" --sticky --message "'$CMD_DISPLAY' has finished - $RESULT"
    if [ `uname -s` == "Linux" ]; then
        # On Linux, notify via notify-send
        which notify-send && notify-send "'$CMD_DISPLAY' has finished" "$RESULT"

if [[ $# -eq 0 ]]; then 
  echo "Usage notify <command to run>"


if [ $EXIT_CODE -eq 0 ]; then
  echo $RESULT    
  echo $RESULT
  exit $EXIT_CODE

When you want to monitor the completion of a command, simply prefix it with notify.

# Longish command you will forget about
rake clean publish

# Longish command you will be notified of upon completion
notify rake clean publish

A few things I particularly like:

  • I can decide which command to monitor
  • it reports the status (success or failure)

No more excuse for Twitter or other time sinkers.

Name: Emmanuel Bernard
Bio tags: French, Open Source actor, Hibernate, (No)SQL, JCP, JBoss, Snowboard, Economy
Employer: JBoss by Red Hat
Resume: LinkedIn
Team blog: in.relation.to
Personal blog: No relation to
Microblog: Twitter, Google+
Geoloc: Paris, France